[pfx] Re: Potential bug in milter interface, can't change first header field

[Wietse Venema via Postfix-users]

Taco de Wolff via Postfix-users:
> Thanks Wietse and Steffen, I forgot to mention that I'm using Postfix
> 3.5.8, but it appears the bug is thus still present in the latest version.
> Looking forward to the fix :-)

Another solution is to adopt Postfix 3.9 (the development release)
where this was fixed two months ago.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: Potential bug in milter interface, can't change first header field

[Steffen Nurpmeso via Postfix-users]

Taco de Wolff via Postfix-users wrote in
 :
 |Thanks Wietse and Steffen, I forgot to mention that I'm using Postfix
 |3.5.8, but it appears the bug is thus still present in the latest version.
 |Looking forward to the fix :-)
 |
 |@Steffen, that is genius and hadn't thought of it. I can confirm that
 |issuing two modifications works as expected. The first prepend the header,
 |and now the chgheader works since it skips the first (recently prepended)
 |header field.
 |
 |insheader index=0 name="From" value="ot...@domain.com"
 |chgheader index=1 name="From" value=""

hm.  Great it works!

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: Potential bug in milter interface, can't change first header field

[Taco de Wolff via Postfix-users]

Thanks Wietse and Steffen, I forgot to mention that I'm using Postfix
3.5.8, but it appears the bug is thus still present in the latest version.
Looking forward to the fix :-)

@Steffen, that is genius and hadn't thought of it. I can confirm that
issuing two modifications works as expected. The first prepend the header,
and now the chgheader works since it skips the first (recently prepended)
header field.

insheader index=0 name="From" value="ot...@domain.com"
chgheader index=1 name="From" value=""

Kind regards,
Taco de Wolff


On Wed, Feb 21, 2024 at 8:01 PM Steffen Nurpmeso via Postfix-users <
postfix-users@postfix.org> wrote:

> Taco de Wolff via Postfix-users wrote in
>  :
>  |While writing a milter for use with Postfix, I was unable to change the
>  |first header field and instead of changing it, Postfix appends it to the
>  |end of the header. Incidentally, as I believed this was a bug with the
>
> insheader with index 0 worked for me two weeks ago with 3.8.5.
> ('Suggesting insheader 0 plus further deletion should, then.
> 'Currently cannot test.)
> There was also a thread on problems with offset 0 a few weeks ago
> regarding such, resulting in a change, but i think this is not
> released yet.
>
> --steffen
> |
> |Der Kragenbaer,The moon bear,
> |der holt sich munter   he cheerfully and one by one
> |einen nach dem anderen runter  wa.ks himself off
> |(By Robert Gernhardt)
> ___
> Postfix-users mailing list -- postfix-users@postfix.org
> To unsubscribe send an email to postfix-users-le...@postfix.org
>
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: Potential bug in milter interface, can't change first header field

[Steffen Nurpmeso via Postfix-users]

Taco de Wolff via Postfix-users wrote in
 :
 |While writing a milter for use with Postfix, I was unable to change the
 |first header field and instead of changing it, Postfix appends it to the
 |end of the header. Incidentally, as I believed this was a bug with the

insheader with index 0 worked for me two weeks ago with 3.8.5.
('Suggesting insheader 0 plus further deletion should, then.
'Currently cannot test.)
There was also a thread on problems with offset 0 a few weeks ago
regarding such, resulting in a change, but i think this is not
released yet.

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: Potential bug in milter interface, can't change first header field

[Wietse Venema via Postfix-users]

The Postfix Milter implementation is sometimes inconsistent about
the "first" header so that it can sometimes not be updated. 

The fix below was in the queue for Postfix 3.5 - 3.8 a few days
before the SMTP smuggling shitshow happened. The last SMTP smuggling
patch was released on January 21. For the sake of everyone's sanity
I will not do more than one stable release per month.

I am cirrently working on a release for Postfix 3.5 - 3.8 that
addresses several old problems and some new ones.

Wietse

20231213

Bugfix (defect introduced: Postfix 2.3): after prepending
a message header with a Postfix access table PREPEND action,
a Milter request to delete or update an existing header
could have no effect, or it could target the wrong instance
of an existing header. Root cause: the fix dated 20141018
for the Postfix Milter client was incomplete. The client
did correctly hide the first, Postfix-generated, Received:
header when sending message header information to a Milter
with the smfi_header() application callback function, but
it was still hiding the first header (instead of the first
Received: header) when handling requests from a Milter to
delete or update an existing header. Problem report by
Carlos Velasco. This change was verified to have no effect
on requests from a Milter to add or insert a header. Files:
cleanup/cleanup_milter.c, cleanup/Makefile.in,
cleanup/test-queue-file18, cleanup/cleanup_milter.in18[a-d],
cleanup/cleanup_milter.ref18
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Potential bug in milter interface, can't change first header field

[Taco de Wolff via Postfix-users]

Hi,

While writing a milter for use with Postfix, I was unable to change the
first header field and instead of changing it, Postfix appends it to the
end of the header. Incidentally, as I believed this was a bug with the
milter library, I rewrote the milter server implementation from scratch and
came to the same conclusion.

Steps to reproduce:

1. Create a file mail.txt with contents:
From: f...@domain.com
To: t...@domain.com
Subject: Test mail

Content.


2. Execute: cat mail.txt > sendmail -t -f f...@domain.com

3. Return modification response in the EOB request as a "chgheader" with
index=1, name="From", value="ot...@domain.com"

4 Verify in /var/spool/mail/[user] that it appends the header field:
>From f...@domain.com Wed Feb 21 21:54:27 2024
Return-Path: 
X-Original-To: r...@domain.com
Delivered-To: r...@domain.com
From: f...@domain.com
To: t...@domain.com
Subject: Test mail
Message-Id: <20240221205427.dfe8187...@domain.com>
Date: Wed, 21 Feb 2024 21:54:27 +0100 (CET)
From: ot...@domain.com

Content.


This will append a header field "From: ot...@domain.com" to the end of the
header fields, instead of replacing the first header field. When you
reverse the order of From and To (i.e. To comes first, and From second) and
you return the same modification response from the milter, it works as
expected; it does not append a new header, but replaces now the second
header field (or equally, the first From header field).

Is this because Postfix expects the first header field to be a "Received"
header field it prepends itself? According to the source code this field
cannot be changed and is skipped when issuing chgheader from a filter (
https://github.com/vdukhovni/postfix/blob/master/postfix/src/cleanup/cleanup_milter.c#L798).
Reading the "/var/spool/mail/root" file I do not see a Received header
prepended for these mails (it does for other mails...!).

Kind regards,
Taco de Wolff
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: Bug in COMPATIBILITY_README

[Wietse Venema via Postfix-users]

Rune Philosof via Postfix-users:
> Mismatching between compatibility_level in overview and explanations for
> http://www.postfix.org/COMPATIBILITY_README.html#relay_restrictions
> and
> http://www.postfix.org/COMPATIBILITY_README.html#smtputf8_enable
> 
> The overview lists them as compatibility_level < 2 and the detailed 
> explanation says < 1.

I have moved smtpd_relay_restrictions and smtputf8_enable under the
right grouping.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Bug in COMPATIBILITY_README

[Rune Philosof via Postfix-users]

Mismatching between compatibility_level in overview and explanations for
http://www.postfix.org/COMPATIBILITY_README.html#relay_restrictions
and
http://www.postfix.org/COMPATIBILITY_README.html#smtputf8_enable

The overview lists them as compatibility_level < 2 and the detailed explanation 
says < 1.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: When using LDAP over socket, „smtpd_sender_login_maps“ requires an LDAP socket relative to chroot in contrast to other map configurations (potential bug?)

[Viktor Dukhovni via Postfix-users]

On Sun, Nov 05, 2023 at 12:13:17PM +, Matthias Nagel via Postfix-users 
wrote:

> Viktor, you recommend to use proxymap in combination with LDAP,

Yes.

> especially if all LDAP lookups use the same connection.

Regardless of whether the connection settings are the same across all
tables.  But if they are, then multiple tables will share connections,
either per-process, or in the proxymap(8) service (really same thing,
only centralised via an "RPC" indirection).


> Indeed, this is the case for my setup. The LDAP server, the bind DN
> and bind passwd are the same. Only the search base, the query filter
> and the result is different depending on the specific lookup.

In that case the connection would typically be shared.  The Postfix
LDAP_README notes:

Multiple LDAP maps share the same LDAP connection if they differ
only in their query related parameters: base, scope, query_filter,
and so on. To take advantage of this, avoid spurious differences in
the definitions of LDAP maps: host selection order, version, bind,
tls parameters, ... should be the same for multiple maps whenever
possible.

> I tried to read the man page for proxymap (8).

There's not much there to read, instead see DATABASE_README, and the
"proxy_read_maps" and "local_recipient_maps" documentation in
postconf(5).  Yes, the description is somewhat terse, but there's not
a lot say.

> Do I understand correctly, that I only have to add „proxy:“ in front
> of all my „ldap:“ lookups and that‘s it?

That's all, provided that nobody made the mistake of enabling chroot
also for proxymap in master.cf.

> No further configuration is required?

Magic... :-)

> Does proxymap somehow cleverly detect if two LDAP queries use the the
> same connection options and then re-uses the same connection?

That's built into the Postfix LDAP driver, regardless of whether LDAP
lookups are initiating in each process independently, or delegated to
proxymap.

> I excpected that I had to configure each connection which I would like
> to run through the proxy with proxymap, but this doesn‘t seem to be
> the case.

The supported tables are configured via proxy_read_maps and default to
all the usual tables.  Customisation is only needed if you add new
"proxy:" tables not tied to one of the "standard" /mumble_maps/
parameters, for example, in smtpd_recipient_restrictions.  Those
tables would need to be added to the definition of proxy_read_maps,
which you'd have to copy and extended (there's not a "+=" syntax
in main.cf for extending default values).

> Wietse, you say that Postfix cannot control when the LDAP client
> library opens a connection to the LDAP server.

Postfix asks the LDAP library to create a logical LDAP connection,
leaving it to the library to actually to the TCP connect and LDAP
"bind" handshake on demand (an actual query) later.

The timing of the LDAP connect is therefore determined by the LDAP
library.  The Postfix LDAP driver may at times ask the LDAP library
to disconnect from a non-responsive server and ask it to reconnect
to any of the defined servers (ideally it will choose another that
is responsive, if multiple servers are defined).

Bottom line, be a copycat (do what I always did, when I used LDAP),
define somewhere in main.cf:

ldap = proxy:ldap:${config_directory}/
cidr = cidr:${config_directory}/
pcre = pcre:${config_directory}/
indexed = ${default_database_type}:${config_directory}/
...

and then when defining LDAP tables, write:

# For all /mumble_maps/:
mumble_maps = ${ldap}config.cf

instead of:

mumble_maps = ldap:/etc/postfix/config.cf

and likewise for other tables:

...
header_checks = ${pcre}header-checks.pcre
smtpd_client_restrictions =
...
check_client_access ${cidr}client-access.cidr
check_sender_access ${indexed}sender-access
...
...

-- 
Viktor.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: When using LDAP over socket, „smtpd_sender_login_maps“ requires an LDAP socket relative to chroot in contrast to other map configurations (potential bug?)

[Wietse Venema via Postfix-users]

As Viktor mentions, best practice is to:

- Share the LDAP socket handle among the three tables that connect
to the same LDAP endpoint (i.e. delay the bind with bind=no in the
three *cf files).

- Open LDAP tables from outside the chroot, by configuring LDAP
tables as proxy:ldap:/path/to/file, and by running the proxymnap
service with chroot turned off. The proxymap service shares each
LDAP table with multiple Postfix SMTP server processes.

As fir you claims about the contents of the three *.cf files,
I won't comment on that without having seen concrete evidence.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: When using LDAP over socket, „smtpd_sender_login_maps“ requires an LDAP socket relative to chroot in contrast to other map configurations (potential bug?)

[Matthias Nagel via Postfix-users]

Dear Viktor, dear Wietse,

Viktor, you recommend to use proxymap in combination with LDAP, especially if 
all LDAP lookups use the same connection. Indeed, this is the case for my 
setup. The LDAP server, the bind DN and bind passwd are the same. Only the 
search base, the query filter and the result is different depending on the 
specific lookup. I tried to read the man page for proxymap (8). Do I understand 
correctly, that I only have to add „proxy:“ in front of all my „ldap:“ lookups 
and that‘s it? No further configuration is required? Does proxymap somehow 
cleverly detect if two LDAP queries use the the same connection options and 
then re-uses the same connection? I excpected that I had to configure each 
connection which I would like to run through the proxy with proxymap, but this 
doesn‘t seem to be the case.

Wietse, you say that Postfix cannot control when the LDAP client library opens 
a connection to the LDAP server, but at the same time you say that difference 
in LDAP client behaviour is caused by differences in Postfix main.cf/master.cf 
settings and differences in Postfix LDAP configuraton files. I can assure you 
that the only difference is the config file to which „ldap:...“ points, i.e. my 
config file contains

virtual_mailbox_maps = ldap:/etc/postfix/ldap/virtual-mailboxes.cf
virtual_alias_maps = ldap:/etc/postfix/ldap/virtual-aliases.cf
smtpd_sender_login_maps = ldap:/etc/postfix/ldap/sender-login.cf

There is not very much else I could do differently in the config file. The 
three LDAP configuration files are identical in terms of host and binding 
settings. They only differ in query filter and result attribute (obviously). 
Nonetheless, Postfix behaves differently with repect to whether it opens the 
LDAP connection before or after chrooting. While it is technically correct that 
the LDAP client library opens the connections, it is still Postfix which calls 
the the client library and obviously it does so for „virtual_mailbox_maps“ and 
„virtual_alias_maps“ before it chroots, but for „smtpd_sender_login_maps“ after 
it chroots. This is something which Postfix can control. If this difference in 
behaviour is not easily fixable or even intended by design, it should at least 
be mentioned in the docs. It caught me by surprise and it also makes „portmap 
-q“ less useful. For „smtpd_sender_login_maps“ the LDAP configuration must be 
written from the chroot perspective which is not handled by „portmap -q“. Here 
is another rather old thread 
(https://groups.google.com/g/list.postfix.users/c/JZxZiOMmgKk) which never got 
an answer. I bet the author encountered the same problem.

Bests, Matthias


Am Samstag, 4. November 2023, 22:33:48 CET schrieb Wietse Venema via 
Postfix-users:
> Viktor Dukhovni via Postfix-users:
> > On Sat, Nov 04, 2023 at 09:48:32AM -0400, Wietse Venema via Postfix-users 
> > wrote:
> > 
> > > To be precise: Postfix opens your LDAP configuration file and asks
> > > the LDAP library to create an LDAP client instance, before entering
> > > the chroot jail and before accepting any SMTP client commmands.
> > > 
> > > HOWEVER, Postfix does not connect to LDAP sockets before entering
> > > the chroot jail and before accepting any SMTP client commmands. The
> > > LDAP library decides when it wants to do that.
> > 
> > IIRC there we were once upon a time requeting immediate connections to
> > LDAP, but that was not ideal:
> > 
> > - It complicated connection sharing across multiple tables with
> >   the same underlying backend server, that differ only in the
> >   query deails.
> > 
> > - It also (when chrooted) meant automatic reconnect on error
> >   to an alternative server, ... would not necessarily work.
> > 
> > - ...
> > 
> > IIRC, the is in principle a way to perform an early, rather than delayed
> > LDAP bind, but the OP should instead use:
> > 
> > proxy:ldap:...
> > 
> > with "proxyread" not chrooted.  This further improves connection sharing
> > and is a best practice.
> 
> Confirmed. proxy:ldap improves sharing and can sidestep chroot issues,
> as long as the read-only 'proxymap' service is not chrooted in master.cf.
> 
>   Wietse
> ___
> Postfix-users mailing list -- postfix-users@postfix.org
> To unsubscribe send an email to postfix-users-le...@postfix.org
> 


-- 
Matthias Nagel
Dachtlerstr. 2, 40499 Stuttgart
Festnetz: +49-711-25295180, Mobil: +49-151-15998774
E-Mail: matthias.h.na...@posteo.de, Skype: nagmat84, Threema: 86VM8KN7
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: When using LDAP over socket, „smtpd_sender_login_maps“ requires an LDAP socket relative to chroot in contrast to other map configurations (potential bug?)

[Wietse Venema via Postfix-users]

Viktor Dukhovni via Postfix-users:
> On Sat, Nov 04, 2023 at 09:48:32AM -0400, Wietse Venema via Postfix-users 
> wrote:
> 
> > To be precise: Postfix opens your LDAP configuration file and asks
> > the LDAP library to create an LDAP client instance, before entering
> > the chroot jail and before accepting any SMTP client commmands.
> > 
> > HOWEVER, Postfix does not connect to LDAP sockets before entering
> > the chroot jail and before accepting any SMTP client commmands. The
> > LDAP library decides when it wants to do that.
> 
> IIRC there we were once upon a time requeting immediate connections to
> LDAP, but that was not ideal:
> 
> - It complicated connection sharing across multiple tables with
>   the same underlying backend server, that differ only in the
>   query deails.
> 
> - It also (when chrooted) meant automatic reconnect on error
>   to an alternative server, ... would not necessarily work.
> 
> - ...
> 
> IIRC, the is in principle a way to perform an early, rather than delayed
> LDAP bind, but the OP should instead use:
> 
> proxy:ldap:...
> 
> with "proxyread" not chrooted.  This further improves connection sharing
> and is a best practice.

Confirmed. proxy:ldap improves sharing and can sidestep chroot issues,
as long as the read-only 'proxymap' service is not chrooted in master.cf.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: When using LDAP over socket, „smtpd_sender_login_maps“ requires an LDAP socket relative to chroot in contrast to other map configurations (potential bug?)

[Viktor Dukhovni via Postfix-users]

On Sat, Nov 04, 2023 at 09:48:32AM -0400, Wietse Venema via Postfix-users wrote:

> To be precise: Postfix opens your LDAP configuration file and asks
> the LDAP library to create an LDAP client instance, before entering
> the chroot jail and before accepting any SMTP client commmands.
> 
> HOWEVER, Postfix does not connect to LDAP sockets before entering
> the chroot jail and before accepting any SMTP client commmands. The
> LDAP library decides when it wants to do that.

IIRC there we were once upon a time requeting immediate connections to
LDAP, but that was not ideal:

- It complicated connection sharing across multiple tables with
  the same underlying backend server, that differ only in the
  query deails.

- It also (when chrooted) meant automatic reconnect on error
  to an alternative server, ... would not necessarily work.

- ...

IIRC, the is in principle a way to perform an early, rather than delayed
LDAP bind, but the OP should instead use:

proxy:ldap:...

with "proxyread" not chrooted.  This further improves connection sharing
and is a best practice.

-- 
Viktor.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: When using LDAP over socket, „smtpd_sender_login_maps“ requires an LDAP socket relative to chroot in contrast to other map configurations (potential bug?)

[Wietse Venema via Postfix-users]

Matthias Nagel via Postfix-users:
> Hello all,
> 
> I am using Postfix 3.8.1 on Ubuntu 23.10. Per distribution default,
> Postfix runs chrooted. I have setup LDAP lookups for most maps.
> OpenLDAP is only listening via UNIX socket on
> ldapi:///var/run/slapd/ldapi.
>
> For all but one LDAP lookup the corresponding Postfix configuration file 
> looks like
> 
> root@h12345678:~ # cat /etc/postfix/ldap/virtual-mailboxes.cf 
> server_host = ldapi:///var/run/slapd/ldapi
> bind = yes
> bind_dn = ...
> bind_pw = ...
> 
> The path to the socket is absolute. Absolute socket paths work for
> "virtual_mailbox_maps", "virtual_alias_maps", etc. and all other
> directive except for ?smtpd_sender_login_maps?.

The Postfix SMTP server opens 'tables' before entering the chroot
jail and before accepting any SMTP client commmands:

smtpd_sender_login_maps, relay_recipient_maps, virtual_mailbox_maps,
virtual_alias_maps, canonical_maps, sender_canonical_maps,
recipient_canonical_maps, local_recipient_maps, relay_clientcerts,
permit_mx_backup_networks, relay_domains, mynetworks, and more.

To be precise: Postfix opens your LDAP configuration file and asks
the LDAP library to create an LDAP client instance, before entering
the chroot jail and before accepting any SMTP client commmands.

HOWEVER, Postfix does not connect to LDAP sockets before entering
the chroot jail and before accepting any SMTP client commmands. The
LDAP library decides when it wants to do that.

IN PARTICULAR if the LDAP client decides to open the LDAP socket
at the time that Postfix makes a query, then the LDAP client won't
have access to pathnames outside the chroot jail.

According to your logs, the LDAP library attempts to 'connect' to
the LDAP server *after* the Postfix SMTP server has entered the
chroot jail and after accepting any SMTP client commmands.

That behavior is outside the control of Postfix.

For more support please follow instructios at
https://www.postfix.org/DEBUG_README.html#mail

Any differences in LDAP client behavior would be caused by:

- Differences in Postfix main.cf/master.cf settings

- Differences in Postfix LDAP configuraton files.

- Differences in SeLinux/AppArmor/anvivirus policies for different
  Postfix processes.

- Other differences that cannot be inferred from your report.

Note that the Postfix LDAP client does not care what lookup table
it is used for; any differences in behavior must necessarily be the
result of differences in Postfix configuration, or differences
external to Postfix.

Wietse
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] When using LDAP over socket, „smtpd_sender_login_maps“ requires an LDAP socket relative to chroot in contrast to other map configurations (potential bug?)

[Matthias Nagel via Postfix-users]

Hello all,

I am using Postfix 3.8.1 on Ubuntu 23.10. Per distribution default, Postfix 
runs chrooted. I have setup LDAP lookups for most maps. OpenLDAP is only 
listening via UNIX socket on ldapi:///var/run/slapd/ldapi.

For all but one LDAP lookup the corresponding Postfix configuration file looks 
like

root@h12345678:~ # cat /etc/postfix/ldap/virtual-mailboxes.cf 
server_host = ldapi:///var/run/slapd/ldapi
bind = yes
bind_dn = ...
bind_pw = ...

The path to the socket is absolute. Absolute socket paths work for 
"virtual_mailbox_maps", "virtual_alias_maps", etc. and all other directive 
except for „smtpd_sender_login_maps“.

"smtpd_sender_login_maps" requires a *relative* path to the LDAP socket and the 
LDAP socket must be located within the chrooted environment. With an absolute 
path I get the following error logs

my-host.my-domain.tld postfix/submission/smtpd[341439]: >>> START Sender 
address RESTRICTIONS <<<
my-host.my-domain.tld postfix/submission/smtpd[341439]: generic_checks: 
name=reject_authenticated_sender_login_mismatch
my-host.my-domain.tld postfix/submission/smtpd[341439]: ctable_locate: leave 
existing entry key jane@receiver.tld?sen...@my-domain.tld
my-host.my-domain.tld postfix/submission/smtpd[341439]: dict_ldap_lookup: In 
dict_ldap_lookup
my-host.my-domain.tld postfix/submission/smtpd[341439]: dict_ldap_lookup: No 
existing connection for LDAP source /etc/postfix/ldap/sender-login.cf, reopening
my-host.my-domain.tld postfix/submission/smtpd[341439]: dict_ldap_connect: 
Connecting to server ldapi:///var/run/slapd/ldapi
my-host.my-domain.tld postfix/submission/smtpd[341439]: dict_ldap_connect: 
Actual Protocol version used is 3.
my-host.my-domain.tld postfix/submission/smtpd[341439]: dict_ldap_connect: 
Binding to server ldapi:///var/run/slapd/ldapi with dn ...
my-host.my-domain.tld postfix/submission/smtpd[341439]: warning: 
dict_ldap_connect: Unable to bind to server ldapi:///var/run/slapd/ldapi with 
dn ...
my-host.my-domain.tld postfix/submission/smtpd[341439]: warning: 
ldap:/etc/postfix/ldap/sender-login.cf lookup error for "sen...@my-domain.tld"
my-host.my-domain.tld postfix/submission/smtpd[341439]: maps_find: 
smtpd_sender_login_maps: sen...@my-domain.tld: search aborted
my-host.my-domain.tld postfix/submission/smtpd[341439]: NOQUEUE: reject: RCPT 
from dial-up.client.provider.tld[x.y.w.z]: 451 4.3.0 : 
Temporary lookup failure

In order to make it work, the configuration file for "smtpd_sender_login_maps" 
must look like

root@h12345678:~ # cat /etc/postfix/ldap/sender-login.cf 
server_host = ldapi://private/ldapi
bind = yes
bind_dn = ...
bind_pw = ...

Note, that there is only a double slash (//) after the protocol specifier, not 
a tripple slash (///) to form a relative path. I also had to make OpenLDAP 
listen on that additional socket (obvisouly). With that modified configuration, 
LDAP lookup for „smtpd_sender_login_maps“ does work.

However, and that is annoying, postmap stops working for this particular map, 
i.e.

postmap -q sen...@my-domain.tld ldap:/etc/postfix/ldap/sender-login.cf

returns an error, because postmap does not chroot postmap does not find the 
LDAP socket.

 - Why does „smtpd_sender_login_maps“ behave differently than all other 
configuration options which allow LDAP lookup?
 - Is this an oversight? Is it an „bug“ in the Postfix software? All other LDAP 
connections seem to opened by Postfix before chrooting.
 - Did I miss something in the docs? If this is not a bug, but intended 
behaviour, there should at least a hint in the docs that 
„smtpd_sender_login_maps“ is special with respect to LDAP configuration

Bests, Matthias


___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: [pfx-dev] Re: Bug in Dovecot SASL driver: authentication failure reason is wrong

[Wietse Venema via Postfix-users]

Stephan Bosch via Postfix-devel:
> 
> Op 2-11-2023 om 15:22 schreef Wietse Venema:
> > Stephan Bosch via Postfix-devel:
> >> Looks like Postfix [...] somehow uses the data from the previous CONT auth 
> >> service
> >> response as the reason.
> > Does this patch address the problem? It resets any previous Dovecot
> > auth service response before parsing the next Dovecot auth server
> > response.
> >
> > Wietse
> 
> It does get rid of the base64 mess:
> 
> 18:43:38.179584 send: 'AUTH OAUTHBEARER =\r\n' 18:43:42.184373 reply: 
> b'334 
> eyJzdGF0dXMiOiJpbnZhbGlkX3Rva2VuIiwib3BlbmlkLWNvbmZpZ3VyYXRpb24iOiJodHRwczovL2lkLm9wZW4teGNoYW5nZS5jb20vb2lkYy9jb25maWcifQ==\r\n'
>  
> 18:43:42.184613 send: 'AQ==\r\n' 18:43:44.188326 reply: b'535 5.7.8 
> Error: authentication failed: \r\n'
> 
> Still, the error is now `Error: authentication failed: `. This also 
> looks wrong, since it just ends in a colon and white space. The absence 
> of a specific error message should be handled specially I guess.

The 'reason' value is now an empty string. What would you suggest instead?

Error: authentication failed\r\n

Error: authentication failed: some other text here\r\n
 
Wietse


___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: BUG: Postfix deals badly with corrected-typo in aliases :(

[Viktor Dukhovni via Postfix-users]

On Sun, Aug 27, 2023 at 04:06:18PM -0400, Viktor Dukhovni via Postfix-users 
wrote:

> If the aliases(5) table has actually been rebuilt, and the message
> is now deliverable, the background refresh is supposed to happen:
> 
> address_verify_negative_refresh_time (default: 3h)
>The time after which a failed address verification probe needs to be
>refreshed.
> 
>Specify a non-zero time value (an integral value plus an optional
>one-letter suffix that specifies the time unit).  Time units: s
>(seconds), m (minutes), h (hours), d (days), w (weeks).  The default
>time unit is h (hours).
> 
>This feature is available in Postfix 2.1 and later.
> 
> 
> https://github.com/vdukhovni/postfix/blob/v3.8.1/postfix/src/verify/verify.c#L512-L550
> 
> If you have detailed log evidence of refresh probes not happening,
> please share.

FWIW, I am running an experiment on my end, asking the verify(8) service
to verify a non-existent virtual mailbox:

#define DEL_RCPT_STAT_OK0
#define DEL_RCPT_STAT_DEFER 1
#define DEL_RCPT_STAT_BOUNCE2
#define DEL_RCPT_STAT_TODO  3

# date; postconf address_verify_negative_refresh_time; printf 
'request\0query\0address\0postfix-lus...@dukhovni.org\0\0' | nc -NU 
/var/spool/postfix/private/verify | tr '\0' '\n'
Sun Aug 27 16:35:39 EDT 2023
address_verify_negative_refresh_time = 1200s
protocol
address_verification_prrotocol

status
0
recipient_status
3
reason
Address verification in progress

The initial query returns a pending (TODO) result, which in short order
resolves to an error (BOUNCE) status:

Aug 27 16:35:39 amnesiac postfix/cleanup[55691]: C8FA712A3DC: 
message-id=<...>
Aug 27 16:35:39 amnesiac postfix/qmgr[55681]: C8FA712A3DC: 
from=, size=273, nrcpt=1 (queue active)
Aug 27 16:35:39 amnesiac postfix/error[55692]: C8FA712A3DC: 
to=, relay=none, delay=0.02, delays=0/0.02/0/0, 
dsn=5.1.1, status=undeliverable (User unknown in virtual alias table)
Aug 27 16:35:39 amnesiac postfix/qmgr[55681]: C8FA712A3DC: removed

# date; postconf address_verify_negative_refresh_time; printf 
'request\0query\0address\0postfix-lus...@dukhovni.org\0\0' | nc -NU 
/var/spool/postfix/private/verify | tr '\0' '\n'
Sun Aug 27 16:36:18 EDT 2023
address_verify_negative_refresh_time = 1200s
protocol
address_verification_prrotocol

status
0
recipient_status
2
reason
User unknown in virtual alias table

After waiting out the 20 minute refresh TTL, another verify(8) query
returns the same result:

# date; postconf address_verify_negative_refresh_time; printf 
'request\0query\0address\0postfix-lus...@dukhovni.org\0\0' | nc -NU 
/var/spool/postfix/private/verify | tr '\0' '\n'
Sun Aug 27 16:55:44 EDT 2023
address_verify_negative_refresh_time = 1200s
protocol
address_verification_prrotocol

status
0
recipient_status
2
reason
User unknown in virtual alias table

However, there's also a background probe:

Aug 27 16:55:44 amnesiac postfix/cleanup[55843]: 9E78612A27D: 
message-id=<...>
Aug 27 16:55:44 amnesiac postfix/qmgr[55681]: 9E78612A27D: 
from=, size=273, nrcpt=1 (queue active)
Aug 27 16:55:44 amnesiac postfix/error[55844]: 9E78612A27D: 
to=, relay=none, delay=0.02, delays=0/0.02/0/0, 
dsn=5.1.1, status=undeliverable (User unknown in virtual alias table)
Aug 27 16:55:44 amnesiac postfix/qmgr[55681]: 9E78612A27D: removed

If I had already added the problem user, this probe would have updated
the database to indicate that the user exists.  After dropping the
refresh time to 120s, and adding the table entry, we see:

# date; postconf address_verify_negative_refresh_time; printf 
'request\0query\0address\0postfix-lus...@dukhovni.org\0\0' | nc -NU 
/var/spool/postfix/private/verify | tr '\0' '\n'
Sun Aug 27 17:02:28 EDT 2023
address_verify_negative_refresh_time = 120s
protocol
address_verification_prrotocol

status
0
recipient_status
2
reason
User unknown in virtual alias table

But now the resulting background probe succeeds:

Aug 27 17:02:28 amnesiac postfix/cleanup[55936]: 29A8112A603: 
message-id=<...>
Aug 27 17:02:28 amnesiac postfix/qmgr[55921]: 29A8112A603: 
from=, size=273, nrcpt=1 (queue active)
Aug 27 17:02:28 amnesiac postfix/virtual[55937]: 29A8112A603: to=<...>, 
orig_to=, relay=virtual, delay=0.03, 
delays=0/0.02/0/0, dsn=2.0.0, status=deliverable (delivers to maildir)
Aug 27 17:02:28 amnesiac postfix/qmgr[55921]: 29A8112A603: removed

So that, shortly after verification returns sucess (OK):

# date; postconf address_verify_negative_refresh_time; printf 
'request\0query\0address\0postfix-lus...@dukhovni.org\0\0' | nc -NU 
/var/spool/postfix/private/verify | tr '\0' '\n'
Sun Aug 27 17:06:08 EDT 2023
address_verify_negative_refresh_time = 120s
protocol

[pfx] Re: BUG: Postfix deals badly with corrected-typo in aliases :(

[Viktor Dukhovni via Postfix-users]

On Sun, Aug 27, 2023 at 01:41:19PM -0600, Pete Holzmann wrote:

> Ummm... Viktor, how many people do *you* think have read the fine
> documentation on every verification option they use in their main.cf 
> restriction configurations?

I don't know.  What I do know is that using features whose documentation
has not been read is liable to lead to naïvely surprising behaviour. :-(

> Beyond that, from experience this statement is false:
> 
> >As documented, but there should also be a background refresh every 3
> >hours, so that the second try 3h after correcting the problem, will
> >succeed: 
> 
> >address_verify_negative_expire_time = 3d
> >address_verify_negative_refresh_time = 3h
> >address_verify_positive_expire_time = 31d
> >address_verify_positive_refresh_time = 7d

If the aliases(5) table has actually been rebuilt, and the message
is now deliverable, the background refresh is supposed to happen:

address_verify_negative_refresh_time (default: 3h)
   The time after which a failed address verification probe needs to be
   refreshed.

   Specify a non-zero time value (an integral value plus an optional
   one-letter suffix that specifies the time unit).  Time units: s
   (seconds), m (minutes), h (hours), d (days), w (weeks).  The default
   time unit is h (hours).

   This feature is available in Postfix 2.1 and later.


https://github.com/vdukhovni/postfix/blob/v3.8.1/postfix/src/verify/verify.c#L512-L550

If you have detailed log evidence of refresh probes not happening,
please share.

> I have 100% default values. Ten hours after correcting the problem, 
> it still failed. Perhaps a bug?

During those 10 hours, how many deliveries were attempted to the
problem address after the source table was updated and the indexed
"database" file rebuilt.  The "refresh" probes only run in response
to delivery attempts, in the background, after returning a cached
response to the SMTP client.

> >Only if you haven't read the documentation.
> 
> Whether or not the fine documentation has been read, no information 
> in the log, even at log-level 10, points to the verification cache.

Well, "log level 10" (whatever that means) is a good way of not seeing
the forest for the trees.  At most one "-v" is generally all that one
might temporarily enable to get more insight into the details of message
processing.  In this case for the smtpd(8) and/or verify(8) services.

> >No, newaliases have nothing to do with this.  This is entirely material
> >for ADDRESS_VERIFICATION_README and verify(8), but perhaps the timer
> >parameters could be also mentioned prominently in the README file. 
> 
> Why would it not make sense for newaliases to trigger a refresh? 
> Clearly, newaliases implies that aliases have been changed, so it is 
> quite possible that prior failures are now successes. ;)

Because there are over 1000 tunable parameters, with many of them
potentially affecting the deliverability of messages to a recipient.
There nothing particularly remarkable about the content of aliases(5).

The behaviour of recipient verification is documented in verify(8) and
ADDRESS_VERIFICATION_README, with per-parameter details in postconf(5).

-- 
Viktor.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Re: BUG: Postfix deals badly with corrected-typo in aliases :(

[Viktor Dukhovni via Postfix-users]

On Fri, Aug 25, 2023 at 08:07:01PM -0600, Pete Holzmann via Postfix-users wrote:

> SUMMARY
> 
> * Scenario/repeatability: 
>- See www.postfix.org/ADDRESS_VERIFICATION_README.html#caching
>- Since Postfix 2.7, there's a persistent verification database.

Actually, there isn't, or, more precisely, it isn't actually used by
default.  You have to choose to enable recipient verification, and if
you do, you have read the fine documentation on address verification,
including the documentation on the cache TTLs in:

http://www.postfix.org/verify.8.html

which is referenced from, e.g.

http://www.postfix.org/ADDRESS_VERIFICATION_README.html#dirty_secret

and various other sections.

>- By default, all failed probes (even of local/aliases) expire after 
>three days.

As documented, but there should also be a background refresh every 3
hours, so that the second try 3h after correcting the problem, will
succeed:

address_verify_negative_expire_time = 3d
address_verify_negative_refresh_time = 3h
address_verify_positive_expire_time = 31d
address_verify_positive_refresh_time = 7d

>- Workaround: postfix stop; rm $data_directory/verify_cache.db; 
>postfix start

Tune the settings to your needs.  There's no need to remove the
database.  That also wipes your positive cache results.

>- This is a very non-intuitive implementation, and very difficult for 
>an average admin to diagnose: no information points to the 
>verification cache.

Only if you haven't read the documentation.

>- Having examined the (verify.c) code, I realize a 'nice' solution 
>(clear cache of local address entries after newaliases) isn't that 
>simple. Someone with more experience can likely design a proper fix.

Reading the code is not necessary.  Just the documentation is
sufficient.

> * Suggestions
>- At the very least, this needs to be highlighted in documentation 
>for newaliases etc

No, newaliases have nothing to do with this.  This is entirely material
for ADDRESS_VERIFICATION_README and verify(8), but perhaps the timer
parameters could be also mentioned prominently in the README file.

-- 
Viktor.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] BUG: Postfix deals badly with corrected-typo in aliases :(

[Pete Holzmann via Postfix-users]

SUMMARY

* Scenario/repeatability: 
   - Typical default postfix configuration
   - An alias is added, but with a typo.
   - Incoming email is rejected as expected, "Recipient address 
   rejected: undeliverable address: unknown user: "address"
   - Typo corrected. newaliases is used. Postfix can be restarted. 
   Doesn't matter.
   - Local email to the alias now works
   - Incoming email is still rejected, as it fails 
   reject_unverified_recipient. Even ten hours later.

* Expected result
   - After newaliases, and particularly after postfix reload, the alias 
   should function.

* Turned on debugging to log level 10. Revealed nothing interesting.

* Analysis
   - See www.postfix.org/ADDRESS_VERIFICATION_README.html#caching
   - Since Postfix 2.7, there's a persistent verification database.
   - By default, address_verify_map =  
   btree:$data_directory/verify_cache
   - By default, the database survives newaliases, postfix reload, and 
   postfix stop
   - By default, all failed probes (even of local/aliases) expire after 
   three days.
   - Workaround: postfix stop; rm $data_directory/verify_cache.db; 
   postfix start

* Discussion

   - This is a very non-intuitive implementation, and very difficult for 
   an average admin to diagnose: no information points to the 
   verification cache.
   - Having examined the (verify.c) code, I realize a 'nice' solution 
   (clear cache of local address entries after newaliases) isn't that 
   simple. Someone with more experience can likely design a proper fix.

* Suggestions
   - At the very least, this needs to be highlighted in documentation 
   for newaliases etc
   - Next step: distinguish local and non-local addresses; enable reset 
   of local cache 
   entries after newaliases.

[end]
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] possible bug in rspamd

[Benny Pedersen via Postfix-users]

ARC-Authentication-Results: i=1; list.sys4.de; dkim=fail;
  arc=none (Message is not ARC signed);
  dmarc=none
Authentication-Results: list.sys4.de; dkim=fail;
 arc=none (Message is not ARC signed); dmarc=none

when originating mails is not dkim signed then i belive dkim=fail here 
should be reported dkim=none, not fail ?

___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

[pfx] Bug Report: main.cf smtpd_client_restrictions check_ccert_access fails when table spec contains spaces inside braces

[Sean Gallagher via Postfix-users]

I have been trying to understand why check_ccert_access does not work 
with an inline:{} table and I believe I have uncovered a subtle bug.


My investigation has focused on 
https://github.com/vdukhovni/postfix/blob/master/postfix/src/global/map_search.c 



To cut to the chase, I believe line 161 should call mystrtokq() instead 
of mystrtok().


Reasoning..

from http://www.postfix.org/postconf.5.html we have the example:
check_ccert_access { type:table, { search_order = cert_fingerprint, 
pubkey_fingerprint } }


from this, lets choose an "inline" table as per 
http://www.postfix.org/DATABASE_README.html
check_ccert_access { inline:{ key=value, { key = text with whitespace or 
comma }}, { search_order = cert_fingerprint, pubkey_fingerprint } }


Now lets see how the parser in map_search.c performs.

We can follow the code to the call:
map_search_create("{ inline:{ key=value, { key = text with whitespace or 
comma }}, { search_order = cert_fingerprint, pubkey_fingerprint } }")


Line 156 detects this is a "Long form" because it starts with a "{"

Line 158 strips away the outermost pair of braces "{" "}" and then trims 
whitespace, leaving:
"inline:{ key=value, { key = text with whitespace or comma }}, { 
search_order = cert_fingerprint, pubkey_fingerprint }"


Line 161 then calls mystrtok() which splits the string at the next 
whitespace separator - WITHOUT REGARD FOR BRACES

So now we have
map_type_name = "inline:{"
bp = "key=value, { key = text with whitespace or comma }}, { 
search_order = cert_fingerprint, pubkey_fingerprint }"


THIS IS CLEARLY NOT WHAT WAS INTENDED

Line 183 expects bp to point to the attribute part, i.e. we want
bp = "{ search_order = cert_fingerprint, pubkey_fingerprint }"

What should have happened is the inline table should have been skimmed 
over by counting opening and closing braces.
This is exactly what the function mystrtokq() does. "q" for "quote" 
presumably.


Repeating the analysis, assuming line 161 calls mystrtokq(), we get
map_type_name = "inline:{ key=value, { key = text with whitespace or 
comma }}"

bp = " { search_order = cert_fingerprint, pubkey_fingerprint }"

Line 183 now gives us:
attr_name_val = "{ search_order = cert_fingerprint, pubkey_fingerprint 
}" and sanity follows.


This one character "q" in the source would have saved many hours of my 
time.



--
This email has been checked for viruses by AVG antivirus software.
www.avg.com___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org

Re: Bug: Postfix errors at startup for service listed in known_tcp_ports but not listed in /etc/services

[Peter]

On 17/10/21 8:00 pm, Viktor Dukhovni wrote:

The feature appears to have been released in an incomplete form.
I don't see any code in Postfix to actually use "known_tcp_ports"
to load the underlying hash table.


Hr, okay.


Also, while numeric service names work with getaddrinfo(3), I
don't believe they work with with getservbyname(3):

 +   if ((sp = getservbyname(filter_known_tcp_port(service), protocol)) 
!= 0)
 +   if ((sp = getservbyname(filter_known_tcp_port(service), protocol)) 
== 0)
 +   if ((sp = getservbyname(filter_known_tcp_port(service), protocol)) 
== 0)
 +if ((sp = getservbyname(filter_known_tcp_port(service), proto)) != 0) 
{

So even when the parameter is properly loaded, only the last two changes
would work as expected.


That makes sense.  It seems like we need a wrapper function for 
getservbyname() that first checks htable_locate() and if the port exists 
there simply creates the servent structure itself with the proper 
service data, and if not passes the call onto getservbyname().



Peter

Re: Bug: Postfix errors at startup for service listed in known_tcp_ports but not listed in /etc/services

[Viktor Dukhovni]

On Sun, Oct 17, 2021 at 06:40:47PM +1300, Peter wrote:

> Just had someone come into the IRC chat with this issue and I was able 
> to reproduce it quite easily, this is with Postfix 3.6.2.  If your 
> /etc/services has smtps listed but not submissions (or vice-versa) and 
> you uncomment or add the relevant section to master.cf then postfix 
> gives an error like the following at startup:
> 
> Oct 17 18:28:59 CentOS8 postfix/master[79810]: fatal: 
> 127.0.0.1:submissions: Servname not supported for ai_socktype
> Oct 17 18:29:00 CentOS8 postfix/master[79809]: fatal: daemon 
> initialization failure
> Oct 17 18:29:01 CentOS8 postfix/postfix-script[79811]: fatal: mail 
> system startup failed

The feature appears to have been released in an incomplete form.
I don't see any code in Postfix to actually use "known_tcp_ports"
to load the underlying hash table.

Also, while numeric service names work with getaddrinfo(3), I
don't believe they work with with getservbyname(3):

--- a/src/posttls-finger/posttls-finger.c
+++ b/src/posttls-finger/posttls-finger.c
@@ -1495 +1496 @@ static char *parse_destination(char *destination, char 
*def_service,
-   if ((sp = getservbyname(service, protocol)) != 0)
+   if ((sp = getservbyname(filter_known_tcp_port(service), protocol)) 
!= 0)
--- a/src/smtp/smtp_connect.c
+++ b/src/smtp/smtp_connect.c
@@ -363 +364 @@ static char *smtp_parse_destination(char *destination, char 
*def_service,
-   if ((sp = getservbyname(service, protocol)) == 0)
+   if ((sp = getservbyname(filter_known_tcp_port(service), protocol)) 
== 0)
--- a/src/util/find_inet.c
+++ b/src/util/find_inet.c
@@ -92 +93 @@ int find_inet_port(const char *service, const char 
*protocol)
-   if ((sp = getservbyname(service, protocol)) == 0)
+   if ((sp = getservbyname(filter_known_tcp_port(service), protocol)) 
== 0)
--- a/src/util/myaddrinfo.c
+++ b/src/util/myaddrinfo.c
@@ -284 +285 @@ static int find_service(const char *service, int socktype)
-if ((sp = getservbyname(service, proto)) != 0) {
+if ((sp = getservbyname(filter_known_tcp_port(service), proto)) != 0) {
@@ -447 +448 @@ int hostname_to_sockaddr_pf(const char *hostname, int 
pf,
-err = getaddrinfo(hostname, service, , res);
+err = getaddrinfo(hostname, filter_known_tcp_port(service), , 
res);
@@ -563 +564 @@ int hostaddr_to_sockaddr(const char *hostaddr, const 
char *service,
-err = getaddrinfo(hostaddr, service, , res);
+err = getaddrinfo(hostaddr, filter_known_tcp_port(service), , 
res);

So even when the parameter is properly loaded, only the last two changes
would work as expected.

-- 
Viktor.

Bug: Postfix errors at startup for service listed in known_tcp_ports but not listed in /etc/services

[Peter]

Just had someone come into the IRC chat with this issue and I was able 
to reproduce it quite easily, this is with Postfix 3.6.2.  If your 
/etc/services has smtps listed but not submissions (or vice-versa) and 
you uncomment or add the relevant section to master.cf then postfix 
gives an error like the following at startup:


Oct 17 18:28:59 CentOS8 postfix/master[79810]: fatal: 
127.0.0.1:submissions: Servname not supported for ai_socktype
Oct 17 18:29:00 CentOS8 postfix/master[79809]: fatal: daemon 
initialization failure
Oct 17 18:29:01 CentOS8 postfix/postfix-script[79811]: fatal: mail 
system startup failed


...the thing is that submissions is listed in known_tcp_ports:

known_tcp_ports = lmtp=24, smtp=25, smtps=submissions=465, submission=587

...so by rights postfix should know that submissions is port 465 and 
startup without error in spite of the service not being listed in 
/etc/services.


Is there a stray startup check that was missed when known_tcp_ports was 
added or something?



Peter

Re: [BUG] tls_server_sni_maps parameter can't use texthash tables

[Viktor Dukhovni]

On Mon, Jun 14, 2021 at 04:57:24PM -0400, Christopher Gurnee wrote:

> That was quick, thanks!

Welcome to Postfix, where we don't let bugs linger and have no (need for
a) bug tracking system, because there are no open bugs.  Bugs are fixed
in near real time, and show up in snaphots and patch releases not long
after.

There is a WISHLIST file in the development snapshot source releases,
that does capture some hypothetical future improvements...

-- 
Viktor.

Re: [BUG] tls_server_sni_maps parameter can't use texthash tables

[Wietse Venema]

Christopher Gurnee:
> That was quick, thanks!

Well you did the work by finding the missing code in postmap.c :-)

Wietse

Re: [BUG] tls_server_sni_maps parameter can't use texthash tables

[Christopher Gurnee]

That was quick, thanks!

Regards,
Chris

On 06/14/2021 4:22 pm, Wietse Venema wrote:

Wietse Venema:

Christopher Gurnee:
> Workaround
> --
>
> Use a hash table:
>  tls_server_sni_maps = hash:/etc/postfix/tls_server_sni
> and create it with:
>  sudo postmap -F /etc/postfix/tls_server_sni

There is some code that was added to postmap/postmap.c but not to
util/dict_thash.c. This is a bit more than I can do while
reading email. Expect to see some patch in a day or so.


This should work.

Wietse

--- /var/tmp/postfix-3.7-20210612/src/util/dict_thash.c 2017-12-27
17:29:45.0 -0500
+++ src/util/dict_thash.c   2021-06-14 16:10:11.323422125 -0400
@@ -180,6 +181,24 @@
 " is this an alias file?", path, lineno);

/*
+	 * Optionally treat the value as a filename, and replace the 
value

+* with the BASE64-encoded content of the named file.
+*/
+   if (dict_flags & DICT_FLAG_SRC_RHS_IS_FILE) {
+   VSTRING *base64_buf;
+   char   *err;
+
+   if ((base64_buf = dict_file_to_b64(dict, value)) == 0) {
+   err = dict_file_get_error(dict);
+   msg_warn("%s, line %d: %s: skipping this entry",
+VSTREAM_PATH(fp), lineno, err);
+   myfree(err);
+   continue;
+   }
+   value = vstring_str(base64_buf);
+   }
+
+   /*
 * Store the value under the key. Handle duplicates
 * appropriately. XXX Move this into dict_ht, but 1) that map
 * ignores duplicates by default and we would have to check that

Re: [BUG] tls_server_sni_maps parameter can't use texthash tables

[Wietse Venema]

Wietse Venema:
> Christopher Gurnee:
> > Workaround
> > --
> > 
> > Use a hash table:
> >  tls_server_sni_maps = hash:/etc/postfix/tls_server_sni
> > and create it with:
> >  sudo postmap -F /etc/postfix/tls_server_sni
> 
> There is some code that was added to postmap/postmap.c but not to
> util/dict_thash.c. This is a bit more than I can do while
> reading email. Expect to see some patch in a day or so.

This should work.

Wietse

--- /var/tmp/postfix-3.7-20210612/src/util/dict_thash.c 2017-12-27 
17:29:45.0 -0500
+++ src/util/dict_thash.c   2021-06-14 16:10:11.323422125 -0400
@@ -180,6 +181,24 @@
 " is this an alias file?", path, lineno);
 
/*
+* Optionally treat the value as a filename, and replace the value
+* with the BASE64-encoded content of the named file.
+*/
+   if (dict_flags & DICT_FLAG_SRC_RHS_IS_FILE) {
+   VSTRING *base64_buf;
+   char   *err;
+
+   if ((base64_buf = dict_file_to_b64(dict, value)) == 0) {
+   err = dict_file_get_error(dict);
+   msg_warn("%s, line %d: %s: skipping this entry",
+VSTREAM_PATH(fp), lineno, err);
+   myfree(err);
+   continue;
+   }
+   value = vstring_str(base64_buf);
+   }
+
+   /*
 * Store the value under the key. Handle duplicates
 * appropriately. XXX Move this into dict_ht, but 1) that map
 * ignores duplicates by default and we would have to check that

Re: [BUG] tls_server_sni_maps parameter can't use texthash tables

[Wietse Venema]

Christopher Gurnee:
> Workaround
> --
> 
> Use a hash table:
>  tls_server_sni_maps = hash:/etc/postfix/tls_server_sni
> and create it with:
>  sudo postmap -F /etc/postfix/tls_server_sni

There is some code that was added to postmap/postmap.c but not to
util/dict_thash.c. This is a bit more than I can do while
reading email. Expect to see some patch in a day or so.

Wietse

[BUG] tls_server_sni_maps parameter can't use texthash tables

[Christopher Gurnee]

Hi, all! My apologies if I've gotten anything wrong below.

Version
---
3.4.13-0ubuntu1 (from Ubuntu 20.04.2)
(although I suspect this affects all versions >= 3.4)

Configuration
-

/etc/postfix/main.cf:
smtpd_tls_security_level = may
tls_server_sni_maps = texthash:/etc/postfix/tls_server_sni

/etc/postfix/tls_server_sni:
mail.example.net /etc/letsencrypt/live/mail.example.net/privkey.pem,
 
/etc/letsencrypt/live/mail.example.net/fullchain.pem


Expected


When smtpd receives STARTTLS with an SNI of mail.example.net, TLS should 
start.


What actually happens
-

When smtpd receives STARTTLS with an SNI of mail.example.net, this is 
logged:


postfix/smtpd[1994543]: warning: table 
texthash:/etc/postfix/tls_server_sni: key mail.example.net: malformed 
BASE64 value: /etc/letsencrypt/live/mail.exa
postfix/smtpd[1994543]: warning: tls_server_sni_maps: 
mail.example.net map lookup problem
postfix/smtpd[1994543]: SSL_accept error from 
mail.example.com[10.10.10.10]: -1
postfix/smtpd[1994543]: warning: TLS library problem: 
error:1422E0EA:SSL routines:final_server_name:callback 
failed:../ssl/statem/extensions.c:1006:
postfix/smtpd[1994543]: lost connection after STARTTLS from 
mail.example.com[10.10.10.10]
postfix/smtpd[1994543]: disconnect from 
mail.example.com[10.10.10.10] ehlo=1 starttls=0/1 commands=1/2


Workaround
--

Use a hash table:
tls_server_sni_maps = hash:/etc/postfix/tls_server_sni
and create it with:
sudo postmap -F /etc/postfix/tls_server_sni

Details
---

According to the docs, smtpd should read the tls_server_sni texthash 
file on startup, then read the two .pem files, base64-encode their 
concatenated contents, and store that base64 as the value for the 
"mail.example.net" key. Based on the error message above it appears to 
be skipping the base64 steps, and when it receives STARTTLS and should 
be base64-decoding the file contents, it instead attempts to 
base64-decode the incorrectly-stored '/etc/letsencrypt/live/mail.exa...' 
string.


You can see here where postmap does some syntax checking, and then 
correctly deals with the base64 case, and stores the results:


https://github.com/vdukhovni/postfix/blob/c0e76bc1f16273f24a4820020f0958d62804a4f2/postfix/src/postmap/postmap.c#L512-L544

I think (not sure) the issue is in dict_thash.c. It does that same 
syntax checking, but despite the earlier comment "This duplicates the 
parser in postmap.c", it skips the base64 case before storing the 
results:


https://github.com/vdukhovni/postfix/blob/c0e76bc1f16273f24a4820020f0958d62804a4f2/postfix/src/util/dict_thash.c#L170-L183

In any case, thanks for taking a look!

Regards,
Chris

Re: bug in debian10

[IL Ka]

It is just a warning, you can live with it.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=926331

On Mon, Oct 26, 2020 at 7:59 PM natan  wrote:

> Hi
> Probably bug in debian 10 ...
> "warning: symlink leaves directory: /etc/postfix/./makedefs.out"
>
> ii  postfix3.4.14-0+deb10u1 amd64High-performance mail
> transport agent
>
> Maybe another repo ? I don't want to install from source ... eh
>
> I search in google and probably bug in
> --
>
>

bug in debian10

[natan]

Hi
Probably bug in debian 10 ...
"warning: symlink leaves directory: /etc/postfix/./makedefs.out"

ii  postfix    3.4.14-0+deb10u1 amd64    High-performance mail
transport agent

Maybe another repo ? I don't want to install from source ... eh

I search in google and probably bug in
--

Re: Minor logrotate bug (was Re: logrotate script for Postfix)

[Wietse Venema]

Larry Stone:
> I posted the note below Saturday but having seen no acknowledgement from 
> Wietse, wanted to make sure it wasn?t overlooked although it?s very minor.
> 
> To summarize what?s below, the default value of maillog_file_rotate_suffix is 
> %Y%M%d-%H%M%S which puts minutes where you expect month. It should be 
> %Y%m%d-%H%M%S.
> 
> And as I said, it?s technically not a bug because it works as documented but 
> as documented and as works is not what one would reasonably expect.
> 
> I?m running Postfix 3.5.1.

I fixed this in the pending stable release updates, and it's
been available the on-line manpages and unstable releases.

Wietse

Minor logrotate bug (was Re: logrotate script for Postfix)

[Larry Stone]

I posted the note below Saturday but having seen no acknowledgement from 
Wietse, wanted to make sure it wasn’t overlooked although it’s very minor.

To summarize what’s below, the default value of maillog_file_rotate_suffix is 
%Y%M%d-%H%M%S which puts minutes where you expect month. It should be 
%Y%m%d-%H%M%S.

And as I said, it’s technically not a bug because it works as documented but as 
documented and as works is not what one would reasonably expect.

I’m running Postfix 3.5.1.

-- 
Larry Stone
lston...@stonejongleux.com





> On May 9, 2020, at 10:31 AM, Larry Stone  wrote:
> 
>> 
>> On May 9, 2020, at 9:45 AM, Wietse Venema  wrote:
>> 
>> 
>> If the log is written by Postfix you must use "postfix logrotate".
>> This ensures that Postfix stops writing to a file before it is
>> compressed.
>> 
>>  Wietse
> 
> I hate to even suggest I found a bug with Postfix, but I think I found a very 
> minor bug.
> 
> First, despite having gone to Postfix logging over a year ago (thanks to 
> MacOS’s weird logging system), this is the first I heard there was a Postfix 
> logrotate command. Testing it, I did not get the rotated file name I would 
> have expected. The bug is the default name for the rotated file which is from 
> the parameter maillog_file_rotate_suffix:
> # postconf -d maillog_file_rotate_suffix
> maillog_file_rotate_suffix = %Y%M%d-%H%M%S
> 
> This is putting minutes where month should be. And it’s documented that way 
> at http://www.postfix.org/MAILLOG_README.html (so technically not a bug since 
> it works as documented but not as one would expect).
> 
> Easy fix with an override in main.cf
> 
> -- 
> Larry Stone
> lston...@stonejongleux.com

Re: probably bug in postfix3-3.4

[Viktor Dukhovni]

On Wed, May 06, 2020 at 09:07:45AM -0400, Wietse Venema wrote:

> You need to collect logs over at least 5 minutes.

I was sitting this one out, but perhaps should make a few comments:

1.  Postfix does not have bugs of the sort conjectured by the OP.
The problem is entirely a result of configuration and workload.

2.  I don't recall seeing the OP's configuration (verbatim, without
refolding of lines, "postconf -nf" and "postconf -Mf" output)
posted to the list.

3.  Frequent routine "postfix reload" is a bad idea.  On queue
manager restart moves all active messages back into the
deferred queue for reprocessing, this is disruptive.
Don't do that.

4.  The queue manager suspends requesting delivery of new mail when
all the transport delivery agents are still busy attempting
to deliver previously scheduled deliveries.

5.  The queue manager also suspends delivery to a transport when
the delivery reports fatal errors.

6.  The queue manager suspends delivery to a particular nexthop
destination when enough back to back deliveries to that
destination fail (not just invalid recipients but failure
to complete the mail transaction).

7.  The most likely reason for queue manager congestion is excessive
bounces to invalid sender addreses, that clog the smtp delivery
agent output queue, and take a long time to time out.

This could be caused missing input recipient validation, or by
misconfiguration that creates an excessive load bounces.  Mail
routing loops can also contribute.

Of course misconfiguration that causes delivery agent failure
or lots of traffic to a non-working destination also "help".

8.  To understand what's in your quuee, see "QSHAPE_README".
Look at both "qshape" and "qshape -s" (the latter is grouped
by sender domain).

http://www.postfix.org/QSHAPE_README.html

If that does not immediately make the problem clear, post the
output and come back and ask for help to interpret it.

9.  As mentioned by others the OP will actually need to show some
logs that involve delivery agents succeeding and failing.
Restart Postfix if necessary, but capture enough logging to
show processing by one of:

smtp  unix  -   -   n   -   -   smtp
relay unix  -   -   n   -   -   smtp
error unix  -   -   n   -   -   error
retry unix  -   -   n   -   -   error
discard   unix  -   -   n   -   -   discard
local unix  -   n   n   -   -   local
virtual   unix  -   n   n   -   -   virtual
lmtp  unix  -   -   n   -   -   lmtp

which write logs tagged as "postfix/", where
 is the last column above.  Note that's "smtp"
not "smtpd".

-- 
Viktor.

Re: probably bug in postfix3-3.4

[Matus UHLAR - fantomas]

natan maciej milaszewski:

Thenx for replay:

May? 5 06:00:51 smtp1 postfix/smtpd[5939]: warning: Illegal address
syntax from unknown[217.153.30.34] in RCPT command: <>

...

May? 5 06:00:54 smtp1 postfix/smtpd[6444]: warning:
unknown[111.72.195.23]: SASL LOGIN authentication failed: authentication
failure
May? 5 06:00:54 smtp1 postfix/submission/smtpd[6464]: warning: hostname
zg-0428c-286.stretchoid.com does not resolve to address 162.243.138.183:
Name or service not known

nothing else


On 06.05.20 09:07, Wietse Venema wrote:

That is FOUR SECONDS of Postfix logging. That us even less
than the Postfix timeout for delivering mail over SMTP.

You need to collect logs over at least 5 minutes.


ideally, check logs between reload and when you notice postfix not running.

mail that enters queue active and qmgr that fints is there is expected.
the question is why nothing happened to the mail later.

see one some of queue ids in logs:

May  6 14:14:45 server postfix/smtpd[10544]: connect from XXX[10.x.x.x]
May  6 14:14:45 server postfix/smtpd[10544]: 56BD5280282: client=XXX[10.x.x.x]
May  6 14:14:45 server postfix/cleanup[10678]: 56BD5280282: 
message-id=<1588767287@xxx.sk>
May  6 14:14:45 server postfix/qmgr[2545]: 56BD5280282: from=, 
size=1035, nrcpt=2 (queue active)
May  6 14:14:45 server postfix/smtpd[10544]: disconnect from XXX[10.x.x.x] 
ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
May  6 14:14:45 server postfix/smtp[10680]: 56BD5280282: to=, 
relay=YYY[y.y.y.y]:25, delay=0.14, delays=0.07/0/0.06/0.01, dsn=2.0.0, status=sent 
(250 2.0.0 046CEjeo032470-046CEjeq032470 Message accepted for delivery)
May  6 14:14:45 server postfix/qmgr[2545]: 56BD5280282: removed


--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I wonder how much deeper the ocean would be without sponges.

Re: probably bug in postfix3-3.4

[Wietse Venema]

natan maciej milaszewski:
> Thenx for replay:
> 
> May? 5 06:00:51 smtp1 postfix/smtpd[5939]: warning: Illegal address
> syntax from unknown[217.153.30.34] in RCPT command: <>
...
> May? 5 06:00:54 smtp1 postfix/smtpd[6444]: warning:
> unknown[111.72.195.23]: SASL LOGIN authentication failed: authentication
> failure
> May? 5 06:00:54 smtp1 postfix/submission/smtpd[6464]: warning: hostname
> zg-0428c-286.stretchoid.com does not resolve to address 162.243.138.183:
> Name or service not known
> 
> nothing else

That is FOUR SECONDS of Postfix logging. That us even less
than the Postfix timeout for delivering mail over SMTP.

You need to collect logs over at least 5 minutes.

Wietse

Re: probably bug in postfix3-3.4

[Peter]

On 5/05/20 5:04 pm, natan maciej milaszewski wrote:

Hi
I have a centos 7 and postfix3-3.4.7-1.gf.el7.x86_64

I reload postfix via:
postfix reload

May  5 06:00:52 smtp1 postfix/master[22162]: reload -- version 3.4.7,
configuration /etc/postfix

And new mail was only added to queue active
They did not want to leave and the queue was growing

May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwf3mj4z4D6T:
from=, size=80199, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQx260b4z4D8d:
from=, size=835317, nrcpt=1 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwh3Cxjz4D7f:
from=, size=80307, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQxP2LwLz4D9N:
from=, size=27942, nrcpt=1 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwf6SQPz4D6d:
from=, size=80210, nrcpt=2 (queue active)

Problem was fixed after restart postfix

I tested reload ~4 times and the problem happened again


For second test i relod postfix via systemd (service postfix reload) -
works fine


I do find this strange because all that systemd does on reload is call 
this command:

/usr/sbin/postfix reload

...which is basically the same command that you're running.  Try 
explicitly running /usr/sbin/postfix reload, though, just in case 
there's another script getting in the way for you, or an alias or 
function.  It might also pay to look into your environment, certain 
environment variables and settings will differ between the environment 
that systemd has when it issues the reload and the one in your shell. 
You can also try:

bash --noprofile --norc -c '/usr/sbin/postfix reload'

Can you check to make sure you're root when you issue postfix reload? 
try temporarily turning off selinux with:

setenforce 0

...don't forget to turn it back on:
setenforce 1

...double check /var/log/maillog in the latest version of postfix, make 
sure to record any log entries that show up at the exact time you run 
"postfix reload".  There should at least be some log entry that shows 
postfix reloading:
May  6 20:23:04 CentOS7 postfix/postfix-script[1789]: refreshing the 
Postfix mail system
May  6 20:23:04 CentOS7 postfix/master[1618]: reload -- version 3.5.0, 
configuration /etc/postfix


Compare your output to what you get with service postfix reload and see 
if there's any difference.


You may also be able to see the logs with:
service postfix status

Is the problem still in the latest version of postfix if you enable the 
gf-testing repo and update?

yum --enablerepo=gf-testing update postfix3\*

Also double check that the postfix install itself has not been corrupted:
rpm -V postfix3

...if you get no output, or only output that reflects changes to config 
files, then the install is fine.


Wietse already pointed you at the DEBUG_README.  In addition to the 
section that he pointed you to, have a look through the rest to see what 
you can do.  Specifically the sections titled "Manually tracing a 
Postfix daemon process" and below that.  Do the traces with both reload 
commands and compare the output between them.


Let me know what you find so I can fix the packages from Ghettoforge in 
case it has something to do with the package builds.  Also if all of the 
above doesn't reveal the issue I can provide you with RPMs that don't 
have the usual set of CentOS patches applied for you to try, then we can 
troubleshoot from there.


Good luck and let me know how you get on.


Peter

Re: probably bug in postfix3-3.4

[natan maciej milaszewski]

Thenx for replay:

May  5 06:00:51 smtp1 postfix/smtpd[5939]: warning: Illegal address
syntax from unknown[217.153.30.34] in RCPT command: <>
May  5 06:00:51 smtp1 postfix/smtpd[6242]: warning: Illegal address
syntax from unknown[217.153.30.34] in RCPT command: <>
May  5 06:00:51 smtp1 postfix/smtpd[6240]: warning: Illegal address
syntax from unknown[217.153.30.34] in RCPT command: <>
May  5 06:00:51 smtp1 postfix/smtpd[5710]: warning: Illegal address
syntax from unknown[217.153.30.34] in RCPT command: <>
May  5 06:00:52 smtp1 postfix/submission/smtpd[6303]: warning: hostname
zg-0428c-286.stretchoid.com does not resolve to address 162.243.138.183:
Name or service not known
May  5 06:00:52 smtp1 postfix/submission/smtpd[6319]: warning: hostname
zg-0428c-286.stretchoid.com does not resolve to address 162.243.138.183:
Name or service not known
May  5 06:00:52 smtp1 postfix/smtps/smtpd[6194]: warning: hostname
host66-188-252-69.limes.com.pl does not resolve to address
188.252.69.66: Name or service not known
May  5 06:00:52 smtp1 postfix/smtpd[6240]: warning:
unknown[45.143.223.91]: SASL LOGIN authentication failed: authentication
failure
May  5 06:00:53 smtp1 postfix/smtpd[6444]: warning: hostname
net6-ip6.linkbg.com does not resolve to address 87.246.7.6: Name or
service not known
May  5 06:00:53 smtp1 postfix/submission/smtpd[6457]: warning: hostname
zg-0428c-286.stretchoid.com does not resolve to address 162.243.138.183:
Name or service not known
May  5 06:00:53 smtp1 postfix/smtps/smtpd[6199]: warning: Illegal
address syntax from static-81-219-84-50.devs.futuro.pl[81.219.84.50] in
RCPT command: <'basia@[...].pl'>
May  5 06:00:53 smtp1 postfix/smtpd[6240]: warning:
unknown[89.111.132.76]: SASL LOGIN authentication failed: authentication
failure
May  5 06:00:53 smtp1 postfix/smtpd[6444]: warning: unknown[87.246.7.6]:
SASL LOGIN authentication failed: authentication failure
May  5 06:00:53 smtp1 postfix/smtps/smtpd[6197]: warning: SASL
authentication failure: Password verification failed
May  5 06:00:53 smtp1 postfix/smtps/smtpd[6197]: warning:
171-103-138-78.static.asianet.co.th[171.103.138.78]: SASL PLAIN
authentication failed: authentication failure
May  5 06:00:53 smtp1 postfix/submission/smtpd[6464]: warning: hostname
zg-0428c-286.stretchoid.com does not resolve to address 162.243.138.183:
Name or service not known
May  5 06:00:54 smtp1 postfix/smtpd[6466]: warning:
unknown[45.143.223.91]: SASL LOGIN authentication failed: authentication
failure
May  5 06:00:54 smtp1 postfix/smtps/smtpd[6199]: warning: Illegal
address syntax from static-81-219-84-50.devs.futuro.pl[81.219.84.50] in
RCPT command: <'biuro@[..].'>
May  5 06:00:54 smtp1 postfix/smtps/smtpd[6471]: warning: hostname
host66-188-252-69.limes.com.pl does not resolve to address
188.252.69.66: Name or service not known
May  5 06:00:54 smtp1 postfix/smtps/smtpd[5897]: warning: Illegal
address syntax from unknown[46.29.149.182] in RCPT command:
<'anna.m@[...].pl'>
May  5 06:00:54 smtp1 postfix/smtpd[6444]: warning:
unknown[111.72.195.23]: SASL LOGIN authentication failed: authentication
failure
May  5 06:00:54 smtp1 postfix/submission/smtpd[6464]: warning: hostname
zg-0428c-286.stretchoid.com does not resolve to address 162.243.138.183:
Name or service not known

nothing else


On 05.05.2020 19:40, Wietse Venema wrote:
> natan maciej milaszewski:
>> Hi
>> I not found any errors:
> RUN THE COMMAND DESCRIBED IN http://www.postfix.org/DEBUG_README.html#logging
>
> $ egrep '(warning|error|fatal|panic):' /some/log/file | more
>
>   Wietse

Re: probably bug in postfix3-3.4

[Wietse Venema]

natan maciej milaszewski:
> Hi
> I not found any errors:

RUN THE COMMAND DESCRIBED IN http://www.postfix.org/DEBUG_README.html#logging

$ egrep '(warning|error|fatal|panic):' /some/log/file | more

Wietse

Re: probably bug in postfix3-3.4

[natan maciej milaszewski]

Hi
I not found any errors:

May  5 06:00:52 smtp1 postfix/master[22162]: reload -- version 3.4.7,
configuration /etc/postfix
May  5 06:00:52 smtp1 postfix/cleanup[5718]: 49GQxc60ggz4D9D:
message-id=
May  5 06:00:52 smtp1 postfix/qmgr[5678]: 49GQxc60ggz4D9D:
from=, size=67939, nrcpt=1 (queue active)
May  5 06:00:52 smtp1 postfix/submission/smtpd[6305]: disconnect from
unknown[10.10.44.61] ehlo=1 auth=1 mail=1 rcpt=1 data=1 quit=1 commands=6
May  5 06:00:52 smtp1 postfix/anvil[5691]: statistics: max connection
rate 81/60s for (smtp:217.153.30.34) at May  5 06:00:46
May  5 06:00:52 smtp1 postfix/anvil[5691]: statistics: max connection
count 16 for (smtps:217.67.201.19) at May  5 06:00:36
May  5 06:00:52 smtp1 postfix/anvil[5691]: statistics: max cache size
159 at May  5 06:00:52
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwf3mj4z4D6T:
from=, size=80199, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQx260b4z4D8d:
from=, size=835317, nrcpt=1 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwh3Cxjz4D7f:
from=, size=80307, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQxP2LwLz4D9N:
from=, size=27942, nrcpt=1 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwf6SQPz4D6d:
from=, size=80210, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwc1Yh5z4D5p:
from=, size=5213, nrcpt=1 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwf0Ng9z4D6B:
from=, size=85005, nrcpt=1 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwh3f0Cz4D7j:
from=, size=80266, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwh43fbz4D7n:
from=, size=80392, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwg2RCvz4D6r:
from=, size=80346, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwg0FwTz4D6X:
from=, size=80342, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwg5Vtdz4D79:
from=, size=80233, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwf4CQdz4D6V:
from=, size=80227, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQxK1Chqz4D92:
from=, size=1289, nrcpt=1 (queue active)
May  5 06:00:53 smtp1 postfix/smtpd[6444]: warning: hostname
net6-ip6.linkbg.com does not resolve to address 87.246.7.6: Name or
service not known
May  5 06:00:53 smtp1 postfix/smtpd[6444]: connect from unknown[87.246.7.6]
May  5 06:00:53 smtp1 postfix/submission/smtpd[6457]: warning: hostname
zg-0428c-286.stretchoid.com does not resolve to address 162.243.138.183:
Name or service not known
May  5 06:00:53 smtp1 postfix/submission/smtpd[6457]: connect from
unknown[162.243.138.183]
May  5 06:00:53 smtp1 postfix/smtps/smtpd[6199]: warning: Illegal
address syntax from static-81-219-84-50.devs.futuro.pl[81.219.84.50] in
RCPT command: <'ba...@rolfarm.lub.pl'>
May  5 06:00:53 smtp1 postfix/smtpd[6240]: warning:
unknown[89.111.132.76]: SASL LOGIN authentication failed: authentication
failure
May  5 06:00:53 smtp1 postfix/smtpd[6444]: warning: unknown[87.246.7.6]:
SASL LOGIN authentication failed: authentication failure

When I restart postfix all works fine

For test i reload again via "postfix reload" and problem exists

every time after "postfix reload" new mails only added to queue active
and the queue was growing


If i reload via "service postfix reload" problem not exist and mailq is
normal



On 05.05.2020 15:16, Wietse Venema wrote:
> Have a look at the error logs.
> http://www.postfix.org/DEBUG_README.html#logging
>
> Look for obvious signs of trouble
> =
>
> Postfix logs all failed and successful deliveries to a logfile.
>
> * When Postfix uses syslog logging (the default), the file is usually
>   called /var/log/maillog, /var/log/mail, or something similar; the
>   exact pathname is configured in a file called /etc/syslog.conf,
>   /etc/rsyslog.conf, or something similar.
>
> * When Postfix uses its own logging system (see MAILLOG_README),
>   the location of the logfile is configured with the Postfix
>   maillog_file parameter.
>
> When Postfix does not receive or deliver mail, the first order of
> business is to look for errors that prevent Postfix from working
> properly:
>
> % egrep '(warning|error|fatal|panic):' /some/log/file | more
>
> Note: the most important message is near the BEGINNING of the output.
> Error messages that come later are less useful.
>
> The nature of each problem is indicated as follows:
>
> * "panic" indicates a problem in the software itself that only a
>   programmer can fix. Postfix cannot proceed until this is fixed.
>
> * "fatal" is the result of missing files, incorrect permissions,
>   incorrect configuration file settings that you can fix. Postfix
>   cannot proceed until this is fixed.
>
> * "error" reports an error condition. For safety reasons, a Postfix
>   process will terminate when more than 13 of these happen.
>
> * "warning" indicates a non-fatal error. These are 

Re: probably bug in postfix3-3.4

[Wietse Venema]

Have a look at the error logs.
http://www.postfix.org/DEBUG_README.html#logging

Look for obvious signs of trouble
=

Postfix logs all failed and successful deliveries to a logfile.

* When Postfix uses syslog logging (the default), the file is usually
  called /var/log/maillog, /var/log/mail, or something similar; the
  exact pathname is configured in a file called /etc/syslog.conf,
  /etc/rsyslog.conf, or something similar.

* When Postfix uses its own logging system (see MAILLOG_README),
  the location of the logfile is configured with the Postfix
  maillog_file parameter.

When Postfix does not receive or deliver mail, the first order of
business is to look for errors that prevent Postfix from working
properly:

% egrep '(warning|error|fatal|panic):' /some/log/file | more

Note: the most important message is near the BEGINNING of the output.
Error messages that come later are less useful.

The nature of each problem is indicated as follows:

* "panic" indicates a problem in the software itself that only a
  programmer can fix. Postfix cannot proceed until this is fixed.

* "fatal" is the result of missing files, incorrect permissions,
  incorrect configuration file settings that you can fix. Postfix
  cannot proceed until this is fixed.

* "error" reports an error condition. For safety reasons, a Postfix
  process will terminate when more than 13 of these happen.

* "warning" indicates a non-fatal error. These are problems that
  you may not be able to fix (such as a broken DNS server elsewhere
  on the network) but may also indicate local configuration errors
  that could become a problem later.

probably bug in postfix3-3.4

[natan maciej milaszewski]

Hi
I have a centos 7 and postfix3-3.4.7-1.gf.el7.x86_64

I reload postfix via:
postfix reload

May  5 06:00:52 smtp1 postfix/master[22162]: reload -- version 3.4.7,
configuration /etc/postfix

And new mail was only added to queue active
They did not want to leave and the queue was growing

May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwf3mj4z4D6T:
from=, size=80199, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQx260b4z4D8d:
from=, size=835317, nrcpt=1 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwh3Cxjz4D7f:
from=, size=80307, nrcpt=2 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQxP2LwLz4D9N:
from=, size=27942, nrcpt=1 (queue active)
May  5 06:00:52 smtp1 postfix/qmgr[6439]: 49GQwf6SQPz4D6d:
from=, size=80210, nrcpt=2 (queue active)

Problem was fixed after restart postfix

I tested reload ~4 times and the problem happened again


For second test i relod postfix via systemd (service postfix reload) -
works fine

Any idea ? maby bug ?

anyone can confirm ?

Re: Bug report: problem with smtp_mx_address_limit = 0

[Luc Pardon]

On 3/04/19 17:13, Viktor Dukhovni wrote:
> 
> 
> It seems you're in a sardonic mood, ... best to not go there.

If I gave that impression, I apologize.

All I was trying to say is that, because of the bug, you may be more
likely to run out of disk space by trying to send to bona fide domains
like postfix.org than because of overworked delivery agents, simply
because MX's with both A and  records are much more common (and
growing) than malicious ones.

Anyway, the most important thing is that the bug has been noted, and
will eventually be fixed, one way or another.

Thanks for that.

Luc

Re: Bug report: problem with smtp_mx_address_limit = 0

[Viktor Dukhovni]

> On Apr 3, 2019, at 7:02 AM, Luc Pardon  wrote:
> 
>> ... and watch their mail queues melt down.
> 
> ... because of all the mail that will get stuck in the queue with
> "server unavailable or unable to receive mail" for _any_ MX that has
> _both_ A and  records.

No, because failing deliveries to domains with tens to hundreds of
unavailable MX host IPs would DoS the queue by tying up delivery
agents for an unreasonably long time.

It seems you're in a sardonic mood, ... best to not go there.
Your bug report has been noted, and the solution will most
likely be to drop the interpretation of 0 == unlimited.  If
someone decides to set an explicit non-default limit, it will
be more obviously a potential resource issue than "I can't
think of what the limit should be, so just turn it off".

-- 
Viktor.

Re: Bug report: problem with smtp_mx_address_limit = 0

[Luc Pardon]

On 3/04/19 01:16, Wietse Venema wrote:

> I prefer to remove the ability to disable safety mechanisms. 

And in your initial response, you also wrote:

> Probably better to not allow a limit-less smtp_mx_address_limit,
> as it makes Postfix vulnerable to resource exhaustion attack.

Both responses seem to translate to "I prefer to remove the ability to
set smtp_mx_address_limit to zero".

As I said, that's fine with me.


My only comment is that this won't bring you much.

As you indicate yourself, people would still have the ability to
effectively disable the safety mechanism (by setting
smtp_mx_address_limit to one bazillion or so).

The only difference is that, if they really want to get rid of the
limit, they would (have to) disable it in an undocumented way.

So, given that you can't prevent people from shooting themselves in the
foot anyway, it seems more logical to let them do it in the (currently)
documented way.


This being said, I personally don't really care whether the ability to
set smtp_mx_address_limit to zero is fixed or removed, but something has
to be done.

As it stands now, people can still set it to zero ...

>
> ... and watch their mail queues melt down.
>

... because of all the mail that will get stuck in the queue with
"server unavailable or unable to receive mail" for _any_ MX that has
_both_ A and  records.

One of each is enough to trigger the bug, even if they would all respond
if Postfix bothered to try. But it doesn't. It simply defers the mail
until it eventually expires and bounces.



Anyway, I have reported the issue and am happy to leave it at that.

Luc

Re: Bug report: problem with smtp_mx_address_limit = 0

[Wietse Venema]

Luc Pardon:
> On the same topic: what if smtp_mx_address_limit was simply made to
> apply for each family separately? E.g. the default of 5 would mean: keep
> max 5 IPv6 addresses _and_ max 5 IPv4's ?

The purpose of these and other Postfix limits is not to frustrate
legitimate mail operators. Instead the purpose is to maintain sanity
under insane conditions. Therefore I would not be comfortable with
suddenly doubling these limits.

I prefer to remove the ability to disable safety mechanisms. If
sites want to receive MX responses that specify a bazillion hosts
(none of which is likely to respond), then they can configure Postfix
to try all those hosts, and watch their mail queues melt down.

Wietse

Re: Bug report: problem with smtp_mx_address_limit = 0

[Luc Pardon]

On 2/04/19 13:21, Wietse Venema wrote:
> Probably better to not allow a limit-less smtp_mx_address_limit,
> as it makes Postfix vulnerable to resource exhaustion attack.
> 
>   Wietse
> 

Fair enough, but then the docs for smtp_mx_address_limit ought to be
changed to remove the "or zero (no limit)" at the end.

I'd also suggest an explicit warning in case it is still set to zero,
instead of the rather misleading (and inaccurate) "server unavailable or
unable to receive mail" for any MX with both A and  records.



On the same topic: what if smtp_mx_address_limit was simply made to
apply for each family separately? E.g. the default of 5 would mean: keep
max 5 IPv6 addresses _and_ max 5 IPv4's ?

That would a) eliminate the need for balancing the mix, b) simplify the
code, and c) allow the client to honor the smtp_address_preference
setting in a safe way.

Separate limit parameters for each family would also do the job, but
that would probably be overkill.

Luc

Re: Bug report: problem with smtp_mx_address_limit = 0

[Wietse Venema]

Probably better to not allow a limit-less smtp_mx_address_limit,
as it makes Postfix vulnerable to resource exhaustion attack.

Wietse

Bug report: problem with smtp_mx_address_limit = 0

[Luc Pardon]

According to the docs, the smtp_mx_address_limit parameter determines
"the maximal number of MX (mail exchanger) IP addresses that can result
from mail exchanger lookups, or zero (no limit)".

However, when setting it to zero, the SMTP client won't even attempt to
deliver to a server that has _both_ IPv4 _and_ IPv6 addresses. Instead,
the mail is deferred forever (and 4 days ;-) with the error message
"4.4.4 server unavailable or unable to receive mail".

Yet a Perl script can connect to all these addresses just fine, and even
deliver mail when talking SMTP to the other side.



The problem seems to be that the function "smtp_balance_inet_proto" (in
smtp_addr.c) does not treat "0" as the special "no limit" case. Instead,
it treats it literally as a limit of zero IP's.

The function contains the line:

  if (v4_count > 0 && v6_count > 0 && v4_count + v6_count > addr_limit)

and inside the "if" block, a clever algorithm attempts to create a mix
of IPv4 and IPv6 addresses, by eliminating some of each kind, while
enforcing the limit.

But when addr_limit is set to 0, all IP's are eliminated from the list.



I confirmed that by adding "-v" to the smtp client in master.cf and
sending an e-mail to an address in a domain with one MX that has one A
and one .

The Postfix logs show that both IP's are found in DNS :

"begin  address list"
"pref 300 host /
"pref 300 host /
"end  address list"

The logs then continue with the tell-tale entry:

"v6_target=0, v4_target=0"

Followed by the two lines:

"begin smtp_balance_inet_proto result address list"
"end smtp_balance_inet_proto result address list"

With nothing in between...


Note that, in the above, I have obfuscated the host name and IP
addresses. I think I could safely post them here, but I suppose that
they won't be needed. Besides, postfix.org has 3 A's and 3 's and
could probably be used as a test case as well ;-)



Adding "addr_limit > 0" as the first condition in the "if" statement
above would likely take care of the problem.

IMHO, a more elegant (because more readable) solution would be to put an
explicit extra test in front of the (already long) "if", like this:

if (addr_limit == 0) {
return (addr_list);
}

A suitable comment in front would make it even more clear.


I have not provided a patch as this seems such a trivial change. In the
Postfix 3.4.5 source code, the offending "if" statement is at line 409
of smtp_addr.c. Alternatively, you can go straight to the
smtp_balance_inet_proto() function in there.

However, this also means that I haven't tested if my proposed solution
actually works. If somebody tells me that it is not (or may not be)
sufficient, I'm willing to dig further myself.


In any case, a fix is not urgent, since there is an easy workaround, and
that is: "don't RTFM", i.e. leave smtp_mx_address_limit at its default of 5.

And if you really want "no limit", you can set it to a ridiculously high
value, such as 100.



>From the changelog, this seems to have been introduced on 20170505, and
in any case, smtp_addr.c has a timestamp of Dec 27, 2017.

If this has been reported already, I apologize. I did a quick search
only and came up empty. If I missed it, this "bug report" becomes a "bug
reminder" :-)


It goes without saying, I hope, that the other client parameters in
main.cf have to be set to the proper values to trigger the bug (e.g.
smtp_balance_inet_proto = yes).



Luc Pardon

Re: Postfix benchmark: bug or performance regression ?

[Wietse Venema]

Ralf Hildebrandt:
> * Viktor Dukhovni :
> > > On Mar 28, 2019, at 12:03 PM, Wietse Venema  wrote:
> > > 
> > > And thank you for your thorough investigation that helped to narrow
> > > down the root cause: under high traffic conditions, LMTP connections
> > > are cached but never reused, therefore those idle cached connections
> > > are exhausting server resources.
> > 
> > Yes, thanks.  I was similarly impressed by the OP's detailed report.
> > 
> > For the record, in case anyone has not read the entire thread, the issue
> > is only with LMTP over unix-domain sockets.  LMTP over TCP is not affected.
> 
> Ah, excellent. I was already afraid my setup would occasionally suffer
> from this effect.

LMTP over TCP uses the same code paths as SMTP.

Wietse

Re: Postfix benchmark: bug or performance regression ?

[Ralf Hildebrandt]

* Viktor Dukhovni :
> > On Mar 28, 2019, at 12:03 PM, Wietse Venema  wrote:
> > 
> > And thank you for your thorough investigation that helped to narrow
> > down the root cause: under high traffic conditions, LMTP connections
> > are cached but never reused, therefore those idle cached connections
> > are exhausting server resources.
> 
> Yes, thanks.  I was similarly impressed by the OP's detailed report.
> 
> For the record, in case anyone has not read the entire thread, the issue
> is only with LMTP over unix-domain sockets.  LMTP over TCP is not affected.

Ah, excellent. I was already afraid my setup would occasionally suffer
from this effect.

-- 
[*] sys4 AG

https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München
   
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: PATCH: Postfix benchmark: bug or performance regression ?

[Wietse Venema]

Juliana Rodrigueiro:
> On Friday, 29 March 2019 00:49:34 CET Wietse Venema wrote:
> 
> > 
> > This is a patch for LMTP connection reuse over UNIX-domain sockets.
> > 
> > Bugfix (introduced: Postfix 3.0): LMTP connections over
> > UNIX-domain sockets were cached but not reused, due to a
> > cache lookup key mismatch. Therefore, idle cached connections
> > could exhaust LMTP server resources, resulting in two-second
> > pauses between email deliveries. This problem was investigated
> > by Juliana Rodrigueiro. File: smtp/smtp_connect.c.
> > 
> > These diffs are identical except that in Postfix 3.4 some macros
> > were renamed to better match their purpose.
> > 
> > Tested with Postfix 3.5. Please give it a try.
> > 
> > wietse
> 
> There is a small problem in the 3.1-3.3.diff though. I believe the variable 
> "dest" should actually be "path".

Oh, right.

> I quickly tested with this modifications in postfix 3.3.2 and everything 
> worked.
> 
> Thank you a lot for the patch!

Thanks for testing.

Wietse

Re: PATCH: Postfix benchmark: bug or performance regression ?

[Juliana Rodrigueiro]

On Friday, 29 March 2019 00:49:34 CET Wietse Venema wrote:

> 
> This is a patch for LMTP connection reuse over UNIX-domain sockets.
> 
> Bugfix (introduced: Postfix 3.0): LMTP connections over
> UNIX-domain sockets were cached but not reused, due to a
> cache lookup key mismatch. Therefore, idle cached connections
> could exhaust LMTP server resources, resulting in two-second
> pauses between email deliveries. This problem was investigated
> by Juliana Rodrigueiro. File: smtp/smtp_connect.c.
> 
> These diffs are identical except that in Postfix 3.4 some macros
> were renamed to better match their purpose.
> 
> Tested with Postfix 3.5. Please give it a try.
> 
>   wietse

There is a small problem in the 3.1-3.3.diff though. I believe the variable 
"dest" should actually be "path".

I quickly tested with this modifications in postfix 3.3.2 and everything 
worked.

Thank you a lot for the patch!

Julie.

PATCH: Postfix benchmark: bug or performance regression ?

[Wietse Venema]

Wietse Venema:
> Juliana Rodrigueiro:
> > > To get rid of the 2s delays:
> > >
> > > /etc/postfix/main.cf:
> > > lmtp_connection_cache_on_demand = no
> > >
> > > Please let us know if that helps. Meanwhile we can develop a proper fix.
> >
> > And yes, it worked, that helped a lot. Although not as fast as before, but
> > now I see only reasonable delays.
> >
> > Thank you very much for looking into it!!!
> 
> And thank you for your thorough investigation that helped to narrow
> down the root cause: under high traffic conditions, LMTP connections
> are cached but never reused, therefore those idle cached connections
> are exhausting server resources.

This is a patch for LMTP connection reuse over UNIX-domain sockets.

Bugfix (introduced: Postfix 3.0): LMTP connections over
UNIX-domain sockets were cached but not reused, due to a
cache lookup key mismatch. Therefore, idle cached connections
could exhaust LMTP server resources, resulting in two-second
pauses between email deliveries. This problem was investigated
by Juliana Rodrigueiro. File: smtp/smtp_connect.c.

Attached are two diffs: one for Postfix versions 3.0-3.5, and one
for Postfix versions 3.4 and 3.5.

These diffs are identical except that in Postfix 3.4 some macros
were renamed to better match their purpose.

Tested with Postfix 3.5. Please give it a try.

wietse
diff -ur /var/tmp/postfix-3.1.11/src/smtp/smtp_connect.c ./src/smtp/smtp_connect.c
--- /var/tmp/postfix-3.1.11/src/smtp/smtp_connect.c	2015-03-29 15:04:22.0 -0400
+++ ./src/smtp/smtp_connect.c	2019-03-28 19:20:20.0 -0400
@@ -487,6 +487,8 @@
  * the "unix:" prefix.
  */
 smtp_cache_policy(state, path);
+if (state->misc_flags & SMTP_MISC_FLAG_CONN_CACHE_MASK)
+	SET_NEXTHOP_STATE(state, dest);
 
 /*
  * Here we ensure that the iter->addr member refers to a copy of the
@@ -562,6 +564,12 @@
 	msg_panic("%s: unix-domain destination not final!", myname);
 	smtp_cleanup_session(state);
 }
+
+/*
+ * Cleanup.
+ */
+if (HAVE_NEXTHOP_STATE(state))
+	FREE_NEXTHOP_STATE(state);
 }
 
 /* smtp_scrub_address_list - delete all cached addresses from list */
diff -ur /var/tmp/postfix-3.5-20190312/src/smtp/smtp_connect.c ./src/smtp/smtp_connect.c
--- /var/tmp/postfix-3.5-20190312/src/smtp/smtp_connect.c	2018-11-19 19:51:37.0 -0500
+++ ./src/smtp/smtp_connect.c	2019-03-28 18:58:04.0 -0400
@@ -492,6 +492,8 @@
  * the "unix:" prefix.
  */
 smtp_cache_policy(state, path);
+if (state->misc_flags & SMTP_MISC_FLAG_CONN_CACHE_MASK)
+	SET_SCACHE_REQUEST_NEXTHOP(state, path);
 
 /*
  * Here we ensure that the iter->addr member refers to a copy of the
@@ -567,6 +569,12 @@
 	msg_panic("%s: unix-domain destination not final!", myname);
 	smtp_cleanup_session(state);
 }
+
+/*
+ * Cleanup.
+ */
+if (HAVE_SCACHE_REQUEST_NEXTHOP(state))
+	CLEAR_SCACHE_REQUEST_NEXTHOP(state);
 }
 
 /* smtp_scrub_address_list - delete all cached addresses from list */

Re: Postfix benchmark: bug or performance regression ?

[Viktor Dukhovni]

> On Mar 28, 2019, at 12:03 PM, Wietse Venema  wrote:
> 
> And thank you for your thorough investigation that helped to narrow
> down the root cause: under high traffic conditions, LMTP connections
> are cached but never reused, therefore those idle cached connections
> are exhausting server resources.

Yes, thanks.  I was similarly impressed by the OP's detailed report.

For the record, in case anyone has not read the entire thread, the issue
is only with LMTP over unix-domain sockets.  LMTP over TCP is not affected.

-- 
Viktor.

Re: Postfix benchmark: bug or performance regression ?

[Wietse Venema]

Juliana Rodrigueiro:
> > To get rid of the 2s delays:
> >
> > /etc/postfix/main.cf:
> > lmtp_connection_cache_on_demand = no
> >
> > Please let us know if that helps. Meanwhile we can develop a proper fix.
>
> And yes, it worked, that helped a lot. Although not as fast as before, but now
> I see only reasonable delays.
>
> Thank you very much for looking into it!!!

And thank you for your thorough investigation that helped to narrow
down the root cause: under high traffic conditions, LMTP connections
are cached but never reused, therefore those idle cached connections
are exhausting server resources.

Wietse

Re: Postfix benchmark: bug or performance regression ?

[Juliana Rodrigueiro]

On Wednesday, 27 March 2019 20:01:49 CET Viktor Dukhovni wrote:
> On Wed, Mar 27, 2019 at 03:36:28PM +0100, Juliana Rodrigueiro wrote:
> > However, during a benchmark, we realized 3.3.2 was 5 times slower than the
> > version before.
> 
> This is misleading.  Postfix is not 5 times slower, your benchmark
> appears to be measuring the LMTP delivery rate to a single sink
> mailbox, and this artificial work-load may be slower under some
> adverse conditions (below...).
> 

Although the benchmark produces a very large artificial work-load, this 
scenario is still a very valid use case, even more considering that 20 emails 
are enough to clog up deliveries.

The stark difference between the LMTP server (cyrus 2.4) log messages is that 
most deliveries reuse the same session id when running postfix <= 2.11.0:

Mar 28 10:22:22 localdomain lmtpunix[4881]: auditlog: append 
sessionid= mailbox= 
uniqueid=<3aab65ba5c98d020> uid=<28047> 
guid=<521e5e4a35095385881941a77f3035f6087c8942>
Mar 28 10:22:22 localdomain lmtpunix[4881]: Delivered: 
<20190328092221.f2f6...@localdomain.com> to mailbox: user.dst
Mar 28 10:22:22 localdomain lmtpunix[4881]: USAGE dst user: 0.00 sys: 
0.00
Mar 28 10:22:22 localdomain lmtpunix[4926]: auditlog: append 
sessionid= mailbox= 
uniqueid=<3aab65ba5c98d020> uid=<28048> 
guid=
Mar 28 10:22:22 localdomain lmtpunix[4926]: Delivered: 
<2019032809.0044...@localdomain.com> to mailbox: user.dst
...
...
Mar 28 10:22:22 localdomain lmtpunix[4926]: auditlog: append 
sessionid= mailbox= 
uniqueid=<3aab65ba5c98d020> uid=<28060> 
guid=<0dc9e922f8b8cfc01ccc5bfeafa8220812adf8fa>
Mar 28 10:22:22 localdomain lmtpunix[4926]: Delivered: 
<2019032809.0da1...@localdomain.com> to mailbox: user.dst
Mar 28 10:22:22 localdomain lmtpunix[4926]: USAGE dst user: 0.00 sys: 
0.004000
Mar 28 10:22:22 localdomain lmtpunix[6449]: auditlog: append 
sessionid= mailbox= 
uniqueid=<3aab65ba5c98d020> uid=<28061> 
guid=<781d02ef98dde39e03461a60aa18beb858f8a5d2>
Mar 28 10:22:22 localdomain lmtpunix[6449]: Delivered: 
<2019032809.0ea9...@localdomain.com> to mailbox: user.dst


In the other hand, for version >= 2.11.1, the sessions are used once per 
process.

Mar 28 10:04:06 localdomain lmtpunix[4273]: Delivered: 
<20190328090404.7cf8...@localdomain.com> to mailbox: user.dst
Mar 28 10:04:06 localdomain lmtpunix[4273]: USAGE dst user: 0.004000 sys: 
0.00
Mar 28 10:04:06 localdomain lmtpunix[4292]: auditlog: append 
sessionid= mailbox= 
uniqueid=<3aab65ba5c98d020> uid=<28032> 
guid=<165f88d841750f2cc9a59a74e777ada4c884dae3>
Mar 28 10:04:06 localdomain lmtpunix[4292]: Delivered: 
<20190328090404.7dee...@localdomain.com> to mailbox: user.dst
Mar 28 10:04:06 localdomain lmtpunix[4292]: USAGE dst user: 0.00 sys: 
0.00
Mar 28 10:04:06 localdomain lmtpunix[4295]: auditlog: append 
sessionid= mailbox= 
uniqueid=<3aab65ba5c98d020> uid=<28033> 
guid=
...
...
Mar 28 10:04:10 localdomain lmtpunix[4295]: auditlog: append 
sessionid= mailbox= 
uniqueid=<3aab65ba5c98d020> uid=<28041> 
guid=<7ba5808ec13795785ab5a5737cfef97ff24d307a>
Mar 28 10:04:10 localdomain lmtpunix[4295]: Delivered: 
<20190328090404.87ac...@localdomain.com> to mailbox: user.dst
Mar 28 10:04:10 localdomain lmtpunix[4295]: USAGE dst user: 0.00 sys: 
0.004000
Mar 28 10:04:12 localdomain lmtpunix[4273]: auditlog: append 
sessionid= mailbox= 
uniqueid=<3aab65ba5c98d020> uid=<28042> 
guid=<8ee41c08e69e9cdf9be61b2f2bad1a2d2c590f12>
Mar 28 10:04:12 localdomain lmtpunix[4273]: Delivered: 
<20190328090404.88ae...@localdomain.com> to mailbox: user.dst

> > Mar 27 14:46:58 localdomain postfix/lmtp[24750]: 7CFDE72:
> > to=, orig_to=,
> > relay=localdomain.com[/var/ imap/socket/lmtp], delay=8.1,
> > delays=0/6/2/0.02, dsn=2.1.5, status=sent (250
> This delivery spent 6s waiting in the active queue and 2s to complete
> a connection.  Why are connections to the LMTP server so expensive?
> Is your syslog service logging synchronously?  Perhaps logging of
> connection establishment (in the LMTP server) is causing congestion
> at LMTP connect?  What is the LMTP server doing in connect and LHLO
> processing?
> 

Unlikely, the LMTP server doesn't log any more or less than it did before.

> > -#define HAVE_NEXTHOP_STATE(state)
> > (STR((state)->iterator->request_nexthop) != 0)
> > +#define HAVE_NEXTHOP_STATE(state)
> > (STR((state)->iterator->request_nexthop)[0] != 0)
> That's a necessary bugfix, without it the test is always true.
> 

I understand the necessary fix, and that is why I was puzzled by the behaviour 
change.

> > Questions:
> > 
> > (1) Does it mean that the lmtp connections via socket are never cached or
> > reused?
> There is no caching by nexthop domain, since there is no nexthop
> domain for unix-domain LMTP sockets.  All there is is a nexthop
> socket address.  So session caching, if any, would naturally be
> just by endpoint address.  However, in 2.11 the smtp_connect_local()
> function also changed:
> 
>
> 

Re: Postfix benchmark: bug or performance regression ?

[Wietse Venema]

Wietse Venema:
> Juliana Rodrigueiro:
> > Excerpt of maillog version > 2.11.1:
> > Mar 27 14:46:50 localdomain postfix/lmtp[24750]: 6CEFF61: 
> > to=, orig_to=, 
> > relay=localdomain.com[/var/
> > imap/socket/lmtp], delay=0.02, delays=0.01/0/0.01/0, dsn=2.1.5, status=sent 
> > (250 2.1.5 Ok SESSIONID=)
> > Mar 27 14:46:50 localdomain postfix/lmtp[24752]: 6E8CE63: 
> > to=, orig_to=, 
> > relay=localdomain.com[/var/
> > imap/socket/lmtp], delay=0.02, delays=0/0/0.01/0.01, dsn=2.1.5, status=sent 
> > (250 2.1.5 Ok SESSIONID=)
> > Mar 27 14:46:50 localdomain postfix/qmgr[24339]: 6E8CE63: removed
> 
> The a/b/c/d delays logging shows:
> a = time from message arrival to last active queue entry
> b = time from last active queue entry to connection setup
> c = time in connection setup, including DNS, EHLO and STARTTLS
> d = time in message transmission
> 
> Here the delays are 0.01s or less.
> 
> > Mar 27 14:46:52 localdomain postfix/lmtp[24752]: 764A06A: 
> > to=, orig_to=, 
> > relay=localdomain.com[/var/
> > imap/socket/lmtp], delay=2, delays=0/2/0/0, dsn=2.1.5, status=sent (250 
> > 2.1.5 
> > Ok SESSIONID=)
> 
> The message had to wait 2s until a delivery agent became available, for
> reasons that will become clear later.
> 
> > Mar 27 14:46:54 localdomain postfix/lmtp[24750]: 778DC61: 
> > to=, orig_to=, 
> > relay=localdomain.com[/var/
> > imap/socket/lmtp], delay=4, delays=0.01/2/2/0.02, dsn=2.1.5, status=sent 
> > (250 
> > 2.1.5 Ok SESSIONID=)
> 
> The message had to wait 2s until a delivery agent became available,
> and then it spent 2s in connection setup (including DNS lookup).
> 
> > Mar 27 14:46:58 localdomain postfix/lmtp[24750]: 7CFDE72: 
> > to=, orig_to=, 
> > relay=localdomain.com[/var/
> > imap/socket/lmtp], delay=8.1, delays=0/6/2/0.02, dsn=2.1.5, status=sent 
> > (250 
> > 2.1.5 Ok SESSIONID=)
> 
> This message had to wait 6s until a delivery agent became available,
> and then it spent 2s in connection setup (including DNS lookup).
> 
> There is a clear pattern in here: under load, the LMTP client spends
> 2s in connection setup (including DNS lookup).
> 

To get rid of the 2s delays:

/etc/postfix/main.cf:
lmtp_connection_cache_on_demand = no

Please let us know if that helps. Meanwhile we can develop a proper fix.

Wietse

Re: Postfix benchmark: bug or performance regression ?

[Viktor Dukhovni]

> On Mar 27, 2019, at 3:01 PM, Viktor Dukhovni  
> wrote:
> 
> There's likely a bug.  We should either simulate a synthetic nexthop
> ($myhostname?) for unix-domain destinations, and then do nexthop
> reuse (and perhaps do no caching by endpoint address for unix-domain
> destinations), or go back to using re-use by endpoint address.
> Currently there's a mismatch, and we're keeping connections alive
> that can't be reused.

And of course the nexthop can't be "$myhostname".  If there are multiple
LMTP unix-domain socket destinations, we must not mix them up, so the
nexthop needs to be just the socket pathname.

-- 
Viktor.

Re: Postfix benchmark: bug or performance regression ?

[Wietse Venema]

Juliana Rodrigueiro:
> Excerpt of maillog version > 2.11.1:
> Mar 27 14:46:50 localdomain postfix/lmtp[24750]: 6CEFF61: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=0.02, delays=0.01/0/0.01/0, dsn=2.1.5, status=sent 
> (250 2.1.5 Ok SESSIONID=)
> Mar 27 14:46:50 localdomain postfix/lmtp[24752]: 6E8CE63: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=0.02, delays=0/0/0.01/0.01, dsn=2.1.5, status=sent 
> (250 2.1.5 Ok SESSIONID=)
> Mar 27 14:46:50 localdomain postfix/qmgr[24339]: 6E8CE63: removed

The a/b/c/d delays logging shows:
a = time from message arrival to last active queue entry
b = time from last active queue entry to connection setup
c = time in connection setup, including DNS, EHLO and STARTTLS
d = time in message transmission

Here the delays are 0.01s or less.

> Mar 27 14:46:52 localdomain postfix/lmtp[24752]: 764A06A: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=2, delays=0/2/0/0, dsn=2.1.5, status=sent (250 2.1.5 
> Ok SESSIONID=)

The message had to wait 2s until a delivery agent became available, for
reasons that will become clear later.

> Mar 27 14:46:54 localdomain postfix/lmtp[24750]: 778DC61: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=4, delays=0.01/2/2/0.02, dsn=2.1.5, status=sent (250 
> 2.1.5 Ok SESSIONID=)

The message had to wait 2s until a delivery agent became available,
and then it spent 2s in connection setup (including DNS lookup).

> Mar 27 14:46:58 localdomain postfix/lmtp[24750]: 7CFDE72: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=8.1, delays=0/6/2/0.02, dsn=2.1.5, status=sent (250 
> 2.1.5 Ok SESSIONID=)

This message had to wait 6s until a delivery agent became available,
and then it spent 2s in connection setup (including DNS lookup).

There is a clear pattern in here: under load, the LMTP client spends
2s in connection setup (including DNS lookup).

Also, the connections are not reused, which may be the result of a
Postfix change that I will look into.

But connection reuse is an optimization; it should never take 2s
to set up an LMTP connection, not even if the LMTP server is on the
other side of the world.

The core problem is that under load, it takes 2s to set up an LMTP
connection. If you can fix that, then Postfix will perform fine
with or without connection reuse.

What kind of LMTP server are you using?

Wietse

Re: Postfix benchmark: bug or performance regression ?

[Viktor Dukhovni]

On Wed, Mar 27, 2019 at 03:36:28PM +0100, Juliana Rodrigueiro wrote:

> However, during a benchmark, we realized 3.3.2 was 5 times slower than the 
> version before.

This is misleading.  Postfix is not 5 times slower, your benchmark
appears to be measuring the LMTP delivery rate to a single sink
mailbox, and this artificial work-load may be slower under some
adverse conditions (below...).

> With version >= 2.11.1, the first emails are received and processed somewhat 
> instantly,

How many?  Are those connections closed or kept open in scache?

> Mar 27 14:46:50 localdomain postfix/qmgr[24339]: 7079468: 
> from=, size=356, nrcpt=1 (queue active)
> Mar 27 14:46:50 localdomain postfix/lmtp[24750]: 6CEFF61: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=0.02, delays=0.01/0/0.01/0, dsn=2.1.5, status=sent 
> (250 2.1.5 Ok SESSIONID=)

> Mar 27 14:46:50 localdomain postfix/lmtp[24752]: 6E8CE63: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=0.02, delays=0/0/0.01/0.01, dsn=2.1.5, status=sent 
> (250 2.1.5 Ok SESSIONID=)

> Mar 27 14:46:52 localdomain postfix/lmtp[24752]: 764A06A: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=2, delays=0/2/0/0, dsn=2.1.5, status=sent (250 2.1.5 
> Ok SESSIONID=)

This message waited 2s in the active queue, before it was its turn
to go, this suggests some other deliveries were still in progress...

> Mar 27 14:46:54 localdomain postfix/lmtp[24750]: 778DC61: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=4, delays=0.01/2/2/0.02, dsn=2.1.5, status=sent (250 

This delivery took 2 seconds to complete a connection, why?

> Mar 27 14:46:58 localdomain postfix/lmtp[24750]: 7CFDE72: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=8.1, delays=0/6/2/0.02, dsn=2.1.5, status=sent (250 

This delivery spent 6s waiting in the active queue and 2s to complete
a connection.  Why are connections to the LMTP server so expensive?
Is your syslog service logging synchronously?  Perhaps logging of
connection establishment (in the LMTP server) is causing congestion
at LMTP connect?  What is the LMTP server doing in connect and LHLO
processing?

> Mar 27 15:00:35 localdomain postfix/lmtp[26451]: AB58261: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], conn_use=2, delay=0.01, delays=0/0/0/0, dsn=2.1.5, 
> status=sent (250 2.1.5 Ok SESSIONID=)

This shows LMTP connection re-use, but that should not make much
difference with unix-domain sockets.  Local connections are supposed
to be fast.

> Mar 27 15:00:35 localdomain postfix/lmtp[26453]: AC80663: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], conn_use=2, delay=0.01, delays=0/0/0/0, dsn=2.1.5, 
> status=sent (250 2.1.5 Ok SESSIONID=)

Again re-use, but the next connection is fresh:

> Mar 27 15:00:35 localdomain postfix/lmtp[26451]: AD4EB67: 
> to=, orig_to=, 
> relay=localdomain.com[/var/
> imap/socket/lmtp], delay=0.01, delays=0/0/0/0, dsn=2.1.5, status=sent (250 
> 2.1.5 Ok SESSIONID=)

> Going deeper with bisecting the code, I found out that the following line 
> created this behaviour change:
> 
> -#define HAVE_NEXTHOP_STATE(state) (STR((state)->iterator->request_nexthop) 
> != 
> 0)
> +#define HAVE_NEXTHOP_STATE(state) 
> (STR((state)->iterator->request_nexthop)[0] != 0)

That's a necessary bugfix, without it the test is always true.

> Questions:
> 
> (1) Does it mean that the lmtp connections via socket are never cached or 
> reused?

There is no caching by nexthop domain, since there is no nexthop
domain for unix-domain LMTP sockets.  All there is is a nexthop
socket address.  So session caching, if any, would naturally be
just by endpoint address.  However, in 2.11 the smtp_connect_local()
function also changed:


https://github.com/vdukhovni/postfix/commit/072fa2d1f7beaf41756c30a69ef96cb611461cec#diff-24650cb98191c311688117b61626fffbR506

to use smtp_reuse_nexthop() rather than smtp_reuse_addr(), so that
SASL authenticated connections might be re-used, but this looks
like a mistake, since we're therefore creating cached sessions that
will never be re-used, but that tie up LMTP server resources,
possibly leading to subsequent congestion under load.

> (2) Is this expected behaviour? Is this a bug?

There's likely a bug.  We should either simulate a synthetic nexthop
($myhostname?) for unix-domain destinations, and then do nexthop
reuse (and perhaps do no caching by endpoint address for unix-domain
destinations), or go back to using re-use by endpoint address.
Currently there's a mismatch, and we're keeping connections alive
that can't be reused.

The fix will be easy, once we decide on the fine details.

-- 
Viktor.

Postfix benchmark: bug or performance regression ?

[Juliana Rodrigueiro]

Hi all!

We used to have postfix 2.6.11 in our systems, which was then updated with no 
problems to 3.3.2.
However, during a benchmark, we realized 3.3.2 was 5 times slower than the 
version before. Even after disabling all mail filters the slowdown was still 
the same.

The benchmark consists of a large amount of emails being fetched and entering 
postfix via the smtpd daemon and then handed over to cyrus via lmtp and socket 
to be delivered mostly to the same recipient.

At first, we thought maybe the concurrency limits have changed, or maybe now 
there are less processes running, but all the configuration variables 
regarding this have not changed.

The benchmark uses 3 emails, but generating 20 emails is sufficient to see 
the bug.

After trials and errors and running out of ideas, I decided to bisect the code 
to see what made this change.

It happens that version 2.11.1 behaves as slow as 3.3.2, and version 2.11.0 as 
fast as 2.6.11. All with the same hardware, same configuration, same load.

With version >= 2.11.1, the first emails are received and processed somewhat 
instantly, after some time it seems that lmtp is not processing anything and 
just waiting while all emails are fetched, so afterwards it starts delivering 
to the recipients at the rate of about 2 per second. (See excerpt of the log 
file)

Excerpt of maillog version > 2.11.1:

Mar 27 14:46:50 localdomain postfix/qmgr[24339]: 7079468: 
from=, size=356, nrcpt=1 (queue active)
Mar 27 14:46:50 localdomain postfix/lmtp[24750]: 6CEFF61: 
to=, orig_to=, relay=localdomain.com[/var/
imap/socket/lmtp], delay=0.02, delays=0.01/0/0.01/0, dsn=2.1.5, status=sent 
(250 2.1.5 Ok SESSIONID=)
Mar 27 14:46:50 localdomain postfix/smtpd[24746]: 718F46A: 
client=localdomain.com[127.0.0.1]
Mar 27 14:46:50 localdomain postfix/qmgr[24339]: 6CEFF61: removed
Mar 27 14:46:50 localdomain postfix/cleanup[24748]: 718F46A: message-
id=<20190327134650.718f...@localdomain.com>
Mar 27 14:46:50 localdomain postfix/qmgr[24339]: 718F46A: 
from=, size=356, nrcpt=1 (queue active)
Mar 27 14:46:50 localdomain postfix/smtpd[24746]: 7273461: 
client=localdomain.com[127.0.0.1]
Mar 27 14:46:50 localdomain postfix/lmtp[24752]: 6E8CE63: 
to=, orig_to=, relay=localdomain.com[/var/
imap/socket/lmtp], delay=0.02, delays=0/0/0.01/0.01, dsn=2.1.5, status=sent 
(250 2.1.5 Ok SESSIONID=)
Mar 27 14:46:50 localdomain postfix/qmgr[24339]: 6E8CE63: removed
...
...
Mar 27 14:46:50 localdomain postfix/cleanup[24748]: 764A06A: message-
id=<20190327134650.764a...@localdomain.com>
Mar 27 14:46:50 localdomain postfix/qmgr[24339]: 764A06A: 
from=, size=356, nrcpt=1 (queue active)
Mar 27 14:46:50 localdomain postfix/smtpd[24746]: 778DC61: 
client=localdomain.com[127.0.0.1]
Mar 27 14:46:50 localdomain postfix/cleanup[24758]: 778DC61: message-
id=<20190327134650.778d...@localdomain.com>
Mar 27 14:46:50 localdomain postfix/qmgr[24339]: 778DC61: 
from=, size=356, nrcpt=1 (queue active)
Mar 27 14:46:50 localdomain postfix/smtpd[24746]: 786A96B: 
client=localdomain.com[127.0.0.1]
Mar 27 14:46:50 localdomain postfix/cleanup[24758]: 786A96B: message-
id=<20190327134650.786a...@localdomain.com>
...
...
Mar 27 14:46:52 localdomain postfix/lmtp[24752]: 764A06A: 
to=, orig_to=, relay=localdomain.com[/var/
imap/socket/lmtp], delay=2, delays=0/2/0/0, dsn=2.1.5, status=sent (250 2.1.5 
Ok SESSIONID=)
Mar 27 14:46:52 localdomain postfix/qmgr[24339]: 764A06A: removed
Mar 27 14:46:54 localdomain postfix/lmtp[24750]: 778DC61: 
to=, orig_to=, relay=localdomain.com[/var/
imap/socket/lmtp], delay=4, delays=0.01/2/2/0.02, dsn=2.1.5, status=sent (250 
2.1.5 Ok SESSIONID=)
Mar 27 14:46:54 localdomain postfix/qmgr[24339]: 778DC61: removed
...
...
Mar 27 14:46:58 localdomain postfix/lmtp[24750]: 7CFDE72: 
to=, orig_to=, relay=localdomain.com[/var/
imap/socket/lmtp], delay=8.1, delays=0/6/2/0.02, dsn=2.1.5, status=sent (250 
2.1.5 Ok SESSIONID=)
Mar 27 14:46:58 localdomain postfix/qmgr[24339]: 7CFDE72: removed



With version <= 2.11.0, the messages are delivered as soon as they arrive.

Excerpt of maillog version <= 2.11.0:

...
...
Mar 27 15:00:35 localdomain postfix/qmgr[26443]: AA6CC67: removed
Mar 27 15:00:35 localdomain postfix/smtpd[26446]: AC80663: 
client=localdomain.com[127.0.0.1]
Mar 27 15:00:35 localdomain postfix/cleanup[26456]: AC80663: message-
id=<20190327140035.ac80...@localdomain.com>
Mar 27 15:00:35 localdomain postfix/qmgr[26443]: AC80663: 
from=, size=356, nrcpt=1 (queue active)
Mar 27 15:00:35 localdomain postfix/lmtp[26451]: AB58261: 
to=, orig_to=, relay=localdomain.com[/var/
imap/socket/lmtp], conn_use=2, delay=0.01, delays=0/0/0/0, dsn=2.1.5, 
status=sent (250 2.1.5 Ok SESSIONID=)
Mar 27 15:00:35 localdomain postfix/smtpd[26446]: AD4EB67: 
client=localdomain.com[127.0.0.1]
Mar 27 15:00:35 localdomain postfix/qmgr[26443]: AB58261: removed
Mar 27 15:00:35 localdomain postfix/cleanup[26456]: AD4EB67: message-
id=<20190327140035.ad4e...@localdomain.com>

Re: Uhm... next bug or my faulty configuration?

[Wietse Venema]

A. Schulze:
> 
> Viktor Dukhovni:
> 
> > Your no-BDAT work-around is sufficient until the code is updated
> > along lines below
> 
> 
> Hello Viktor,
> 
> Thanks for that patch. I confirm it works like expected

Did you test it in smtpd_end_of_data_restrictions?

Wietse

Re: Uhm... next bug or my faulty configuration?

[A. Schulze]

Viktor Dukhovni:


Your no-BDAT work-around is sufficient until the code is updated
along lines below



Hello Viktor,

Thanks for that patch. I confirm it works like expected

Andreas

Old bug: reject_multi_recipient_bounce

[Wietse Venema]

A. Schulze:
> Hello,
> 
> updated from 3.4.1 to 3.4.3 and at the same time dovecot-2.2 to dovecot-2.3 ( 
> + pigeonhole)
> I assume the changes behavior is dovecot/pigeonhole now using the advertised 
> "CHUNKING" extension.
> 
> Now an echo service (dovecot-2.3-pigeonhole) don't send messages anymore.
> Reason: "Data command rejected: Multi-recipient bounce" while there is 
> clearly only one recipient.

This is 13 years old: reject_multi_recipient_bounce has had the same
false rejects in smtpd_end_of_data_restrictions since Postfix 2.2.

Victor's patch addresses the symptom (BDAT) but not the root cause.
The patch below fixes both BDAT and smtpd_end_of_data_restrictions.

In this case, smaller is better.

Wietse

diff -bur /var/tmp/postfix-3.5-20190310/src/smtpd/smtpd_check.c 
./src/smtpd/smtpd_check.c
--- /var/tmp/postfix-3.5-20190310/src/smtpd/smtpd_check.c   2018-08-23 
09:44:18.0 -0400
+++ ./src/smtpd/smtpd_check.c   2019-03-12 08:28:20.627312192 -0400
@@ -4583,7 +4583,7 @@
status = check_recipient_rcpt_maps(state, state->recipient);
} else if (strcasecmp(name, REJECT_MUL_RCPT_BOUNCE) == 0) {
if (state->sender && *state->sender == 0 && state->rcpt_count
-   > (strcmp(state->where, SMTPD_CMD_DATA) ? 0 : 1))
+   > (strcmp(state->where, SMTPD_CMD_RCPT) != 0))
status = smtpd_check_reject(state, MAIL_ERROR_POLICY,
var_mul_rcpt_code, "5.5.3",
"<%s>: %s rejected: Multi-recipient bounce",

Re: Uhm... next bug or my faulty configuration?

[Viktor Dukhovni]

On Mon, Mar 11, 2019 at 11:48:56PM +0100, A. Schulze wrote:

> I assume the changes behavior is dovecot/pigeonhole now using the advertised 
> "CHUNKING" extension.

Yes.

> Reason: "Data command rejected: Multi-recipient bounce" while there is 
> clearly only one recipient.
> 
> Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: 44JCRG5tYPzCqt2:
>  reject: BDAT from signing-milter.org[84.200.211.109]: 550 5.5.3 :
>  Data command rejected: Multi-recipient bounce; from=<> to=
>  proto=ESMTP helo=

> since years I have "smtpd_data_restrictions = 
> reject_multi_recipient_bounce,reject_unauth_pipelining,permit"
> Must that be adjusted with 3.4.x?

Your no-BDAT work-around is sufficient until the code is updated
along lines below:

-- 
Viktor.

--- a/src/smtpd/smtpd.h
+++ b/src/smtpd/smtpd.h
@@ -259,6 +259,10 @@ extern void smtpd_state_reset(SMTPD_STATE *);
 #define SMTPD_CMD_XFORWARD "XFORWARD"
 #define SMTPD_CMD_UNKNOWN  "UNKNOWN"
 
+#define SMTPD_IN_BODY(state) \
+(strcmp((state)->where, SMTPD_CMD_DATA) == 0 || \
+ strcmp((state)->where, SMTPD_CMD_BDAT) == 0)
+
  /*
   * Representation of unknown and non-existent client information. Throughout
   * Postfix, we use the "unknown" string value for unknown client information
--- a/src/smtpd/smtpd_check.c
+++ b/src/smtpd/smtpd_check.c
@@ -4582,8 +4582,8 @@ static int generic_checks(SMTPD_STATE *state, ARGV 
*restrictions,
if (state->recipient && *state->recipient)
status = check_recipient_rcpt_maps(state, state->recipient);
} else if (strcasecmp(name, REJECT_MUL_RCPT_BOUNCE) == 0) {
-   if (state->sender && *state->sender == 0 && state->rcpt_count
-   > (strcmp(state->where, SMTPD_CMD_DATA) ? 0 : 1))
+   if (state->sender && *state->sender == 0
+   && state->rcpt_count > (SMTPD_IN_BODY(state) ? 1 : 0))
status = smtpd_check_reject(state, MAIL_ERROR_POLICY,
var_mul_rcpt_code, "5.5.3",
"<%s>: %s rejected: Multi-recipient bounce",

Uhm... next bug or my faulty configuration?

[A. Schulze]

Hello,

updated from 3.4.1 to 3.4.3 and at the same time dovecot-2.2 to dovecot-2.3 ( + 
pigeonhole)
I assume the changes behavior is dovecot/pigeonhole now using the advertised 
"CHUNKING" extension.

Now an echo service (dovecot-2.3-pigeonhole) don't send messages anymore.
Reason: "Data command rejected: Multi-recipient bounce" while there is clearly 
only one recipient.

the relevant debug logs:

Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 220 signing-milter.org ESMTP 
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: < 
signing-milter.org[84.200.211.109]: EHLO signing-milter.org
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250-signing-milter.org
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250-PIPELINING
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250-SIZE 128000
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250-ENHANCEDSTATUSCODES
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250-8BITMIME
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250-DSN
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250-SMTPUTF8
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250 CHUNKING
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: < 
signing-milter.org[84.200.211.109]: MAIL FROM:<>
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250 2.1.0 Ok
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: < 
signing-milter.org[84.200.211.109]: RCPT TO:
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 250 2.1.5 Ok
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: < 
signing-milter.org[84.200.211.109]: BDAT 882 LAST
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: >>> START Data 
command RESTRICTIONS <<<
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: generic_checks: 
name=reject_multi_recipient_bounce
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: 44JCRG5tYPzCqt2: 
reject: BDAT from signing-milter.org[84.200.211.109]: 550 5.5.3 : Data 
command rejected: Multi-recipient bounce; from=<> to= 
proto=ESMTP helo=
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: generic_checks: 
name=reject_multi_recipient_bounce status=2
Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: >>> END Data command 
RESTRICTIONS <<<

Mar 11 23:27:54 dili postfix-smo/submission/smtpd[22427]: > 
signing-milter.org[84.200.211.109]: 550 5.5.3 : Data command rejected: 
Multi-recipient bounce, servertime=Mar 11 23:27:54, server=signing-milter.org, 
client=84.200.211.109


current solution: run the smtpd with "smtpd_discard_ehlo_keywords=CHUNKING"

Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 220 signing-milter.org ESMTP 
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: < 
signing-milter.org[84.200.211.109]: EHLO signing-milter.org
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: discarding EHLO 
keywords: CHUNKING
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: match_list_match: 
signing-milter.org: no match
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: match_list_match: 
84.200.211.109: no match
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250-signing-milter.org
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250-PIPELINING
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250-SIZE 128000
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250-ETRN
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250-ENHANCEDSTATUSCODES
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250-8BITMIME
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250-DSN
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250 SMTPUTF8
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: < 
signing-milter.org[84.200.211.109]: MAIL FROM:<>
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250 2.1.0 Ok
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: < 
signing-milter.org[84.200.211.109]: RCPT TO:
Mar 11 23:37:57 dili postfix-smo/submission/smtpd[22846]: > 
signing-milter.org[84.200.211.109]: 250 2.1.5 Ok
Mar 11 23:37:57 dili 

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Lex Scarisbrick]

On Sun, Mar 10, 2019 at 9:41 AM Scott Kitterman 
wrote:
> My preference would be to press on with 3.4 (I don't mind packaging the
bug
> fixes if you don't mind releasing them), but if you are going to withdraw
3.4,
> please do it before next Sunday so I can keep it out of the next Debian
> release.

+1

The RFC 3030 support in Postfix 3.4 is very much a welcome addition for
interoperability with Microsoft.  I understand that's coming at it from
wanting something additive and not about what might be broken as a result.
I'm happy to help if there's any specific testing that would help stabilize
the release.  FWIW, I haven't seen any issues ingressing mail from
Microsoft with Postfix 3.4 to date.

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Wietse Venema]

Daniele Nicolodi:
> On 10/03/2019 15:07, Wietse Venema wrote:
> > You are looking from the "we made improvements" angle. I am looking
> > from the "with hard work, we introduce 1 bug in 1000 lines of new
> > code" angle.
> > 
> > In the TLS library there were 1039 additions and 559 deletions from
> > Postfix 3.3.3 to 3.4.1 (diff -bur --new-file for 'c' and 'h' files
> > only, excluding proxy-related code). That count does not distinguish
> > between low-impact changes that affect only nearby code, and
> > high-impact changes that affect multiple lines, such as global ifdefs.
> > 
> > The changes among those 1600 lines that 'broke' intended functionality
> > are 'easy' to weed out - just wait for people to to report breakages.
> > Such a reactive approach might be acceptable in some projects.
> > 
> > I am concerned about the changes among those 1600 lines that did
> > NOT break intended behavior, but that introduced some other problem.
> > 
> > What is the basis for confidence that no such problems have been
> > introduced, if we obviously missed multiple things that could have
> > been found with simple tests?
> 
> I have no say in the development of Postfix, however I believe another
> interesting question that should be asked is: what is going to make you
> more confident in the releasing those changes later this year with
> Postfix 3.5?  Are there circumstances for which more testing and code
> auditing will happen if the code is not released?

Because we won't be changing 1600 lines in a critical library two
months before the release.

Wietse

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Daniele Nicolodi]

On 10/03/2019 15:07, Wietse Venema wrote:
> You are looking from the "we made improvements" angle. I am looking
> from the "with hard work, we introduce 1 bug in 1000 lines of new
> code" angle.
> 
> In the TLS library there were 1039 additions and 559 deletions from
> Postfix 3.3.3 to 3.4.1 (diff -bur --new-file for 'c' and 'h' files
> only, excluding proxy-related code). That count does not distinguish
> between low-impact changes that affect only nearby code, and
> high-impact changes that affect multiple lines, such as global ifdefs.
> 
> The changes among those 1600 lines that 'broke' intended functionality
> are 'easy' to weed out - just wait for people to to report breakages.
> Such a reactive approach might be acceptable in some projects.
> 
> I am concerned about the changes among those 1600 lines that did
> NOT break intended behavior, but that introduced some other problem.
> 
> What is the basis for confidence that no such problems have been
> introduced, if we obviously missed multiple things that could have
> been found with simple tests?

I have no say in the development of Postfix, however I believe another
interesting question that should be asked is: what is going to make you
more confident in the releasing those changes later this year with
Postfix 3.5?  Are there circumstances for which more testing and code
auditing will happen if the code is not released?

Cheers,
Dan

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Wietse Venema]

Viktor Dukhovni:
> On Sun, Mar 10, 2019 at 12:29:44PM -0400, Wietse Venema wrote:
> 
> > > My preference would be to press on with 3.4 (I don't mind packaging the 
> > > bug 
> > > fixes if you don't mind releasing them), but if you are going to withdraw 
> > > 3.4 
> > > please do it before next Sunday so I can keep it out of the next Debian 
> > > release.
> > 
> > We know of multiple bugs that broke 'desirable functionality' after
> > an overhaul of the TLS stack, and that were kindly brought to the
> > developer's attention by folks like you.
> > 
> > I have to consider the possibility that the same overhaul introduced
> > an equal if not larger number of bugs with 'undesirable functionality',
> > and that these bugs will be found by not-so-kind folks, who will
> > report them only if it helps to promote themselves while at the
> > same time destroying Postfix's good reputation.
> 
> Some of the changes in 3.4 are quite useful when Postfix is compiled
> with OpenSSL 1.1.1, which introduces TLS 1.3.
> 
> My sense is that the situation is not so dire as to warrant retracting
> the release.  The recent changes in 3.4 are:
> 
> - Support for server-side SNI, with new code to load multi-algorithm
>   certificate chain sets.  This also loads the key + cert in a
>   single pass when both are in the same file, now with the correct
>   test for the return value (first report bug) and tests.
> 
> - More diligent tlsproxy(8) checking for compatible parameters in TLS
>   connection re-use.  The tlsproxy connection re-use was added earlier,
>   and the more recent changes just fine-tune context sharing.
> 
> - Removal of support for OpenSSL 1.0.2.  Reduces the attack
>   surface and intoduces no new code.
> 
>   This was however the source of the present (second) reported
>   bug, because an #ifdef guard testing for 1.0.2 or later that
>   should have been removed, was left in place.
> 
> - Consolidation of TLS summary logging, and better logging of
>   TLS 1.3 handshake properties.
> 
> Looking over the various changes again, I think 3.4 is fine.  I'll
> do another code review this week.

You are looking from the "we made improvements" angle. I am looking
from the "with hard work, we introduce 1 bug in 1000 lines of new
code" angle.

In the TLS library there were 1039 additions and 559 deletions from
Postfix 3.3.3 to 3.4.1 (diff -bur --new-file for 'c' and 'h' files
only, excluding proxy-related code). That count does not distinguish
between low-impact changes that affect only nearby code, and
high-impact changes that affect multiple lines, such as global ifdefs.

The changes among those 1600 lines that 'broke' intended functionality
are 'easy' to weed out - just wait for people to to report breakages.
Such a reactive approach might be acceptable in some projects.

I am concerned about the changes among those 1600 lines that did
NOT break intended behavior, but that introduced some other problem.

What is the basis for confidence that no such problems have been
introduced, if we obviously missed multiple things that could have
been found with simple tests?

Wietse

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Viktor Dukhovni]

On Sun, Mar 10, 2019 at 12:29:44PM -0400, Wietse Venema wrote:

> > My preference would be to press on with 3.4 (I don't mind packaging the bug 
> > fixes if you don't mind releasing them), but if you are going to withdraw 
> > 3.4 
> > please do it before next Sunday so I can keep it out of the next Debian 
> > release.
> 
> We know of multiple bugs that broke 'desirable functionality' after
> an overhaul of the TLS stack, and that were kindly brought to the
> developer's attention by folks like you.
> 
> I have to consider the possibility that the same overhaul introduced
> an equal if not larger number of bugs with 'undesirable functionality',
> and that these bugs will be found by not-so-kind folks, who will
> report them only if it helps to promote themselves while at the
> same time destroying Postfix's good reputation.

Some of the changes in 3.4 are quite useful when Postfix is compiled
with OpenSSL 1.1.1, which introduces TLS 1.3.

My sense is that the situation is not so dire as to warrant retracting
the release.  The recent changes in 3.4 are:

- Support for server-side SNI, with new code to load multi-algorithm
  certificate chain sets.  This also loads the key + cert in a
  single pass when both are in the same file, now with the correct
  test for the return value (first report bug) and tests.

- More diligent tlsproxy(8) checking for compatible parameters in TLS
  connection re-use.  The tlsproxy connection re-use was added earlier,
  and the more recent changes just fine-tune context sharing.

- Removal of support for OpenSSL 1.0.2.  Reduces the attack
  surface and intoduces no new code.

  This was however the source of the present (second) reported
  bug, because an #ifdef guard testing for 1.0.2 or later that
  should have been removed, was left in place.

- Consolidation of TLS summary logging, and better logging of
  TLS 1.3 handshake properties.

Looking over the various changes again, I think 3.4 is fine.  I'll
do another code review this week.

-- 
Viktor.

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Viktor Dukhovni]

On Sun, Mar 10, 2019 at 02:34:02PM +, Scott Kitterman wrote:

> This worked just fine until 3.3.2-4 inclusive but since I've upgraded
> my sid system yesterday and Postfix was upgraded to 3.4.1-1 I see:
> 
>   postfix/smtp[15202]: warning: Trust anchor files not supported
>   postfix/smtp[15202]: warning: TLS policy lookup error for 
> [domain.tld]:587/domain.tld: client TLS configuration problem
>   postfix/smtp[15202]: warning: TLS policy lookup for 
> [domain.tld]:587/domain.tld: client TLS configuration problem
>   postfix/smtp[15202]: 8B30018835E3: to=, relay=none, 
> delay=1197, delays=1196/0.82/0.36/0, dsn=4.7.5, status=deferred (client TLS 
> configuration problem)

> diff --git a/src/tls/tls_dane.c b/src/tls/tls_dane.c
> index 93f8e2a5..013426b1 100644
> --- a/src/tls/tls_dane.c
> +++ b/src/tls/tls_dane.c
> @@ -1125,7 +1125,6 @@ TLS_DANE *tls_dane_resolve(unsigned port, const char 
> *proto, DNS_RR *hostrr,
>  
>  int tls_dane_load_trustfile(TLS_DANE *dane, const char *tafile)
>  {
> -#ifdef TRUST_ANCHOR_SUPPORT
>  BIO*bp;
>  char   *name = 0;
>  char   *header = 0;
> @@ -1217,9 +1216,6 @@ int tls_dane_load_trustfile(TLS_DANE *dane, const 
> char *tafile)
>  }
>  /* Some other PEM read error */
>  tls_print_errors();
> -#else
> -msg_warn("Trust anchor files not supported");
> -#endif
>  return (0);
>  }

The proposed patch is correct.  Repeated below without "quoting":

diff --git a/src/tls/tls_dane.c b/src/tls/tls_dane.c
index 93f8e2a5..013426b1 100644
--- a/src/tls/tls_dane.c
+++ b/src/tls/tls_dane.c
@@ -1125,7 +1125,6 @@ TLS_DANE *tls_dane_resolve(unsigned port, const char 
*proto, DNS_RR *hostrr,
 
 int tls_dane_load_trustfile(TLS_DANE *dane, const char *tafile)
 {
-#ifdef TRUST_ANCHOR_SUPPORT
 BIO*bp;
 char   *name = 0;
 char   *header = 0;
@@ -1217,9 +1216,6 @@ int tls_dane_load_trustfile(TLS_DANE *dane, const 
char *tafile)
 }
 /* Some other PEM read error */
 tls_print_errors();
-#else
-msg_warn("Trust anchor files not supported");
-#endif
 return (0);
 }
 

-- 
Viktor.

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Wietse Venema]

Scott Kitterman:
> On Sunday, March 10, 2019 11:11:15 AM Wietse Venema wrote:
> > Scott Kitterman:
> > > I received the bug report/patch below from a Debian user.  I'm somewhat
> > > busy this weekend/week, so I decided to forward it without evaluation
> > > rather than sit on it for a week until I could research it.
> > > 
> > > I attempted to remove the distro specific noise from the report.
> > 
> > Sorry for making you the guinea pig.
> > 
> > I am considering to withdraw Postfix 3.4 and do a proper Postfix
> > 3.5 release as planned later this year. We can't afford having a
> > stable release with bug-of-the-week fixes like we have now.
> 
> I knew I was taking a risk jumping to 3.4 late in our release cycle.
> 
> A week from now when it hits Debian Testing, the user base will grow 
> significantly and we'll get more feedback.
> 
> I guess there weren't enough testers for 3.4 before release.  I don't know 
> that that situation will be better later in the year for 3.5.  From my point 
> of view (I don't know about other distros/OS), it would be somewhat painful to
> stay on 3.3 for the next release at this point, but it'll be a lot harder a 
> week from now.
> 
> My preference would be to press on with 3.4 (I don't mind packaging the bug 
> fixes if you don't mind releasing them), but if you are going to withdraw 3.4 
> please do it before next Sunday so I can keep it out of the next Debian 
> release.

We know of multiple bugs that broke 'desirable functionality' after
an overhaul of the TLS stack, and that were kindly brought to the
developer's attention by folks like you.

I have to consider the possibility that the same overhaul introduced
an equal if not larger number of bugs with 'undesirable functionality',
and that these bugs will be found by not-so-kind folks, who will
report them only if it helps to promote themselves while at the
same time destroying Postfix's good reputation.

Wietse

Re: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Postfix User]

On Sun, 10 Mar 2019 11:41:01 -0400, Scott Kitterman stated:

>On Sunday, March 10, 2019 11:11:15 AM Wietse Venema wrote:
>> Scott Kitterman:  
>> > I received the bug report/patch below from a Debian user.  I'm
>> > somewhat busy this weekend/week, so I decided to forward it
>> > without evaluation rather than sit on it for a week until I could
>> > research it.
>> > 
>> > I attempted to remove the distro specific noise from the report.  
>> 
>> Sorry for making you the guinea pig.
>> 
>> I am considering to withdraw Postfix 3.4 and do a proper Postfix
>> 3.5 release as planned later this year. We can't afford having a
>> stable release with bug-of-the-week fixes like we have now.  
>
>I knew I was taking a risk jumping to 3.4 late in our release cycle.
>
>A week from now when it hits Debian Testing, the user base will grow 
>significantly and we'll get more feedback.
>
>I guess there weren't enough testers for 3.4 before release.  I don't
>know that that situation will be better later in the year for 3.5.
>From my point of view (I don't know about other distros/OS), it would
>be somewhat painful to stay on 3.3 for the next release at this point,
>but it'll be a lot harder a week from now.
>
>My preference would be to press on with 3.4 (I don't mind packaging
>the bug fixes if you don't mind releasing them), but if you are going
>to withdraw 3.4, please do it before next Sunday so I can keep it out
>of the next Debian release.
>
>Although Debian doesn't normally allow it, I've gotten permission from
>the release team to update postfix based on your microreleases (thir
>digit) based on the good history with them fixing relevant bugs with
>minimal regression risk.  It's absolutely not a problem for me to stay
>on the 3.4 path if you're up for it.
>
>Scott K

FreeBSD is using Postfix 3.3 for its stable release and Postfix 3.4 for
its development (experimental) release.

-- 
Postfix User

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[PGNet Dev]

To add a possible data point to the convo,

at least one distro, OpenSUSE, is already toying with apparently poorly 
thought-thru patches (aka, not vetted/source here, from upstream) -- 
e,g, here, 
https://build.opensuse.org/package/view_file/openSUSE:Factory/postfix/postfix-linux45.patch?expand=1 
-- to 'old release' postfix 3.3.3 for Linux5 support.


no particular skin off _my_ nose -- i contentedly build/deploy only from 
Postfix upstream source.


i understand it's 'their' decision, good or bad.  nonetheless, it will 
inevitably generate noise here.

Re: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Postfix User]

On Sun, 10 Mar 2019 11:11:15 -0400 (EDT), Wietse Venema stated:

>Scott Kitterman:
>> I received the bug report/patch below from a Debian user.  I'm
>> somewhat busy this weekend/week, so I decided to forward it without
>> evaluation rather than sit on it for a week until I could research
>> it.
>> 
>> I attempted to remove the distro specific noise from the report.  
>
>Sorry for making you the guinea pig.
>
>I am considering to withdraw Postfix 3.4 and do a proper Postfix
>3.5 release as planned later this year. We can't afford having a
>stable release with bug-of-the-week fixes like we have now.
>
>   Wietse


Wietse, the number of bugs found in stable releases of Postfix is
infinitesimal when compared to other vendors who release products under
the "fix it later" theory.

-- 
Postfix User

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Scott Kitterman]

On Sunday, March 10, 2019 11:11:15 AM Wietse Venema wrote:
> Scott Kitterman:
> > I received the bug report/patch below from a Debian user.  I'm somewhat
> > busy this weekend/week, so I decided to forward it without evaluation
> > rather than sit on it for a week until I could research it.
> > 
> > I attempted to remove the distro specific noise from the report.
> 
> Sorry for making you the guinea pig.
> 
> I am considering to withdraw Postfix 3.4 and do a proper Postfix
> 3.5 release as planned later this year. We can't afford having a
> stable release with bug-of-the-week fixes like we have now.

I knew I was taking a risk jumping to 3.4 late in our release cycle.

A week from now when it hits Debian Testing, the user base will grow 
significantly and we'll get more feedback.

I guess there weren't enough testers for 3.4 before release.  I don't know 
that that situation will be better later in the year for 3.5.  From my point 
of view (I don't know about other distros/OS), it would be somewhat painful to 
stay on 3.3 for the next release at this point, but it'll be a lot harder a 
week from now.

My preference would be to press on with 3.4 (I don't mind packaging the bug 
fixes if you don't mind releasing them), but if you are going to withdraw 3.4, 
please do it before next Sunday so I can keep it out of the next Debian 
release.

Although Debian doesn't normally allow it, I've gotten permission from the 
release team to update postfix based on your microreleases (thir digit) based 
on the good history with them fixing relevant bugs with minimal regression 
risk.  It's absolutely not a problem for me to stay on the 3.4 path if you're 
up for it.

Scott K

Re: Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Wietse Venema]

Scott Kitterman:
> I received the bug report/patch below from a Debian user.  I'm somewhat busy 
> this weekend/week, so I decided to forward it without evaluation rather than 
> sit on it for a week until I could research it.
> 
> I attempted to remove the distro specific noise from the report.

Sorry for making you the guinea pig.

I am considering to withdraw Postfix 3.4 and do a proper Postfix
3.5 release as planned later this year. We can't afford having a
stable release with bug-of-the-week fixes like we have now.

Wietse

Fwd: Bug#924183: postfix: Trust anchor files (tafile=) in TLS policy break secure level email delivery

[Scott Kitterman]

I received the bug report/patch below from a Debian user.  I'm somewhat busy 
this weekend/week, so I decided to forward it without evaluation rather than 
sit on it for a week until I could research it.

I attempted to remove the distro specific noise from the report.

Scott K



Package: postfix
Version: 3.4.1-1
Severity: important
Tags: patch

Hi,

I have entries like this:

  [domain.tld]:587secure tafile=/etc/ssl/certs/Lets-Encrypt-Authority-X3.pem

… in the file referenced by:

  smtp_tls_policy_maps = hash:/etc/postfix/tls_policy

This worked just fine until 3.3.2-4 inclusive but since I've upgraded
my sid system yesterday and Postfix was upgraded to 3.4.1-1 I see:

  postfix/smtp[15202]: warning: Trust anchor files not supported
  postfix/smtp[15202]: warning: TLS policy lookup error for 
[domain.tld]:587/domain.tld: client TLS configuration problem
  postfix/smtp[15202]: warning: TLS policy lookup for 
[domain.tld]:587/domain.tld: client TLS configuration problem
  postfix/smtp[15202]: 8B30018835E3: to=, relay=none, 
delay=1197, delays=1196/0.82/0.36/0, dsn=4.7.5, status=deferred (client TLS 
configuration problem)

This seems to come from src/tls/tls_dane.c. I see that 3.4.0 has
modified this file quite a bit, e.g. these lines were removed:

  #if OPENSSL_VERSION_NUMBER >= 0x100fL && \
 (defined(X509_V_FLAG_PARTIAL_CHAIN) || !defined(OPENSSL_NO_ECDH))
  #define TRUST_ANCHOR_SUPPORT

… and there's only one "#ifdef TRUST_ANCHOR_SUPPORT" left, that guards
the warning I'm seeing. This feels like a leftover of an incomplete
cleanup of the TLS support code that happened in this release, such as
dropping support for OpenSSL 1.0.1.

FWIW the attached patch fixes this problem for me. I don't know if it
can cause any trouble.

From 4d98d0aa5aeb4fbb9941a4239251edfb1537a0e9 Mon Sep 17 00:00:00 2001
From: intrigeri 
Date: Sun, 10 Mar 2019 06:29:25 +
Subject: [PATCH] Drop leftover of obsolete check for trust anchor support.

---
 src/tls/tls_dane.c | 4 
 1 file changed, 4 deletions(-)

diff --git a/src/tls/tls_dane.c b/src/tls/tls_dane.c
index 93f8e2a5..013426b1 100644
--- a/src/tls/tls_dane.c
+++ b/src/tls/tls_dane.c
@@ -1125,7 +1125,6 @@ TLS_DANE *tls_dane_resolve(unsigned port, const char *proto, DNS_RR *hostrr,
 
 int tls_dane_load_trustfile(TLS_DANE *dane, const char *tafile)
 {
-#ifdef TRUST_ANCHOR_SUPPORT
 BIO*bp;
 char   *name = 0;
 char   *header = 0;
@@ -1217,9 +1216,6 @@ int tls_dane_load_trustfile(TLS_DANE *dane, const char *tafile)
 }
 /* Some other PEM read error */
 tls_print_errors();
-#else
-msg_warn("Trust anchor files not supported");
-#endif
 return (0);
 }
 
-- 
2.20.1

Re: AAAA requests on IPv6-disconnected system: bug of feature?

[Scott Kitterman]

On Thursday, May 04, 2017 12:56:05 PM Wietse Venema wrote:
> Marat Khalili:
> > > Postfix from me installs with IPv6 turned off. Complain with your
> > > distributor if they change that.
> > 
> > Indeed default inet_protocols value in my distribution is "all", both in
> > configuration created by install script and when corresponding line is
> > commented out. Do you mean, it is not supposed to be this way?
> 
> I am not talking about the default.
> 
> I am talking about "Postfix from me INSTALLS with ipv6 turned off".

Since this is about Postfix as shipped by Ubuntu and as packaged by Debian, 
the bug tracker for one of those distributions is the appropriate place for 
this discussion, not here.  Apologies for not noticing which distro this was 
about sooner.

Scott K

Re: AAAA requests on IPv6-disconnected system: bug of feature?

[Wietse Venema]

Marat Khalili:
> > Postfix from me installs with IPv6 turned off. Complain with your
> > distributor if they change that.
> 
> Indeed default inet_protocols value in my distribution is "all", both in 
> configuration created by install script and when corresponding line is 
> commented out. Do you mean, it is not supposed to be this way?

I am not talking about the default. 

I am talking about "Postfix from me INSTALLS with ipv6 turned off".

Wietse

Re: AAAA requests on IPv6-disconnected system: bug of feature?

[Viktor Dukhovni]

On Thu, May 04, 2017 at 05:18:55PM +0300, Marat Khalili wrote:

> > Postfix from me installs with IPv6 turned off. Complain with your
> > distributor if they change that.
> 
> Indeed default inet_protocols value in my distribution is "all", both in
> configuration created by install script and when corresponding line is
> commented out. Do you mean, it is not supposed to be this way?

The compiled in default is "all", but the prototype "main.cf" file included
with Postfix sources has "ipv4".

$ postconf -d inet_protocols
inet_protocols = all

$ grep inet_protocols conf/main.cf
inet_protocols = ipv4

On Thu, May 04, 2017 at 05:17:01PM +0300, Marat Khalili wrote:

> > To disable outbound IPv6 in Postfix set "inet_protocols = ipv4".  If you set
> > "inet_protocols" to some other value, then Postfix will do nexthop IPv6 
> > lookups.
>
> What will happen in my current setup if  response suddenly becomes
> non-empty? Will it fail to send the message?

Since IPv6 connections can't/won't complete, all messages will of
course be sent via IPv4.  With the address family not enabled in the
kernel, IPv6 failure will be fast (likely the socket(PF_INET6, ...)
system call will fail) and the only effect is a bit more logging in
some cases and the time it takes to do the  lookups.

-- 
Viktor.

Re: AAAA requests on IPv6-disconnected system: bug of feature?

[Marat Khalili]

Postfix from me installs with IPv6 turned off. Complain with your
distributor if they change that.


Indeed default inet_protocols value in my distribution is "all", both in 
configuration created by install script and when corresponding line is 
commented out. Do you mean, it is not supposed to be this way?



--

With Best Regards,
Marat Khalili

Re: AAAA requests on IPv6-disconnected system: bug of feature?

[Marat Khalili]

To disable outbound IPv6 in Postfix set "inet_protocols = ipv4".  If you set
"inet_protocols" to some other value, then Postfix will do nexthop IPv6 lookups.
What will happen in my current setup if  response suddenly becomes 
non-empty? Will it fail to send the message?


--

With Best Regards,
Marat Khalili

Re: AAAA requests on IPv6-disconnected system: bug of feature?

[Viktor Dukhovni]

> On May 4, 2017, at 4:05 AM, Marat Khalili <m...@rqc.ru> wrote:
> 
> Postfix is installed as forwarder to a fixed relay in a system with
> no IPv6 addresses (disabled system-wide by net.ipv6.conf.*.disable_ipv6
> lines in sysctl). Still, for each message it separately requests both
> A and  records of the relay from DNS, as I verified by tcpdump.
> Is it a bug or feature?

To disable outbound IPv6 in Postfix set "inet_protocols = ipv4".  If you set
"inet_protocols" to some other value, then Postfix will do nexthop IPv6 lookups.

> (Yes I know I can explicitly disable IPv6 in postfix configuration too, but 
> that's not the point.)

Actually, that is the point.  If IPv6 lookups are enabled, then they are 
performed.

> My investigation actually started with the following lines in mail.log, which 
> may or may not be related:
> 
>> May  4 07:52:16 postfix postfix/scache[1518]: statistics: domain lookup 
>> hits=0 miss=3 success=0%
> 
> (3 messages were sent in this interval; there are always 0 hits and 0% 
> success rate.)

This is unrelated to the IPv6 question.

-- 
Viktor.

Re: AAAA requests on IPv6-disconnected system: bug of feature?

[Wietse Venema]

Marat Khalili:
> Postfix is installed as forwarder to a fixed relay in a system with no 
> IPv6 addresses (disabled system-wide by net.ipv6.conf.*.disable_ipv6 
> lines in sysctl). Still, for each message it separately requests both A 
> and  records of the relay from DNS, as I verified by tcpdump. Is it 
> a bug or feature? (Yes I know I can explicitly disable IPv6 in postfix 
> configuration too, but that's not the point.)

Postfix from me installs with IPv6 turned off. Complain with your
distributor if they change that.

Wietse

> My investigation actually started with the following lines in mail.log, 
> which may or may not be related:
> 
> > May  4 07:52:16 postfix postfix/scache[1518]: statistics: domain 
> > lookup hits=0 miss=3 success=0%
> 
> (3 messages were sent in this interval; there are always 0 hits and 0% 
> success rate.)
> 
> I'm using Postfix 3.1.0-3 under Ubuntu 16.04. Relay is secure.emailsrvr.com.
> 
> --
> 
> With Best Regards,
> Marat Khalili
> 

AAAA requests on IPv6-disconnected system: bug of feature?

[Marat Khalili]

Postfix is installed as forwarder to a fixed relay in a system with no 
IPv6 addresses (disabled system-wide by net.ipv6.conf.*.disable_ipv6 
lines in sysctl). Still, for each message it separately requests both A 
and  records of the relay from DNS, as I verified by tcpdump. Is it 
a bug or feature? (Yes I know I can explicitly disable IPv6 in postfix 
configuration too, but that's not the point.)


My investigation actually started with the following lines in mail.log, 
which may or may not be related:


May  4 07:52:16 postfix postfix/scache[1518]: statistics: domain 
lookup hits=0 miss=3 success=0%


(3 messages were sent in this interval; there are always 0 hits and 0% 
success rate.)


I'm using Postfix 3.1.0-3 under Ubuntu 16.04. Relay is secure.emailsrvr.com.

--

With Best Regards,
Marat Khalili

Re: verify_cache.db: No such file or directory - possible Berkeley DB bug ?

[Wietse Venema]

Maurizio Caloro:
> Hello
> 
> >From time to time I see on mail.log the following error message:
> 
> Mar 22 23:29:43 mail postfix/verify[2206]: close database
> /var/lib/postfix/verify_cache.db: No such file or directory (possible
> Berkeley DB bug)

Indeed, the db->close() operation returns an error code with some
Berkeley DB implementations. Postfix is written with great care,
and it pays attention to functions that report errors. In this case
the data is known to be stored safely, therefore db->close() should
not return errors. But it does, and Postfix warns.

This was once a fatal error, but it was changed into a warning
because the fatal error broke Postfix on some Linux systems.

> Please what I need to do here ?

Complain to your vendor. Postfix is only the messenger.

Wietse

verify_cache.db: No such file or directory - possible Berkeley DB bug ?

[Maurizio Caloro]

Hello

>From time to time I see on mail.log the following error message:

Mar 22 23:29:43 mail postfix/verify[2206]: close database
/var/lib/postfix/verify_cache.db: No such file or directory (possible
Berkeley DB bug)

 

I have see found different answer, but I don't know which in further
pursues.

Please what I need to do here ?

 

root@mail:~# ls -la /var/lib/postfix/verify_cache.db

-rw-r--r-- 1 postfix postfix 8192 Mar 22 23:26
/var/lib/postfix/verify_cache.db

 

root@mail:/etc/postfix# postconf -d | grep mail_version

mail_version = 2.11.3

 

Debian 3.16.39-1+deb8u2 (2017-03-07

 

Regards

Mauri

Re: AW: Possible Bug ? postfix 3.1.0-3 fails on mysql table lookup

[John Fawcett]

On 12/18/2016 09:38 PM, John Fawcett wrote:
> On 12/18/2016 02:09 AM, Wietse Venema wrote:
>> What if Postfix made an old-style query? I think it should just
>> report the old-style error here.
>>
>>  Wietse
> I agree. It might be as simple as changing
>
> msg_warn("dict_mysql: stored procedure did not return any result set
> containing data");
>
> to
>
> msg_warn("mysql query failed: %s", mysql_error(host->db));
>
>
> But I think I need to do some tests with unmodified postfix and then compare 
> behaviour to the patched version to make sure the warnings are really the 
> same.
>
> John
>
I have revised the patch and changed the logic. The error messages are
now in line with the exisiting behaviour for queries that return
results, queries that return no results and for queries that have
errors. There is one minor difference in error messages in the case of a
query that contains no SELECT. The result returned by mysql is
indistinguishable from a stored procedure that contains no SELECT and as
I am explicitly setting errno to ENOSUP, the message given for the query
would now be:

postmap -q j...@erba.tv mysql:/etc/postfix/test_update.cf
postmap: warning: mysql query failed:
postmap: fatal: table mysql:/etc/postfix/test_update.cf: query
error: Operation not supported

instead of the existing

postmap -q j...@erba.tv mysql:/etc/postfix/test_update.cf
postmap: warning: mysql query failed:
postmap: fatal: table mysql:/etc/postfix/test_update.cf: query
error: Success

Queries containing errors continue to have errno 0 as currently

postmap -q erba.tv mysql:/etc/postfix/test_query_error.cf
postmap: warning: mysql query failed: Unknown column 'domains' in
'field list'
postmap: fatal: table mysql:/etc/postfix/test_query_error.cf: query
error: Success

though I would opt to set it in this case too.

This version of the patch includes some documentation updates to clarify
that the stored procedure must execute exactly one SELECT statement. The
code has been modified for the revised error reporting and readability.

--- postfix-3.2-20161204_orig/proto/mysql_table2016-10-01
15:44:52.0 +0200
+++ postfix-3.2-20161204/proto/mysql_table2016-12-25
09:04:09.230528031 +0100
@@ -289,6 +289,52 @@
 #certificate.
 # .sp
 #This parameter is available with Postfix 2.11 and later.
+# USING MYSQL STORED PROCEDURES
+# .ad
+# .fi
+#With Postfix 3.2 it is possible to call a stored procedure
+#instead of using a SELECT statement in the query, e.g.
+#
+# .nf
+#query = CALL lookup('%s')
+# .fi
+#
+#The preivously described '%' expansions can be used in the
+#parameter(s) to the stored procedure.
+#
+#The stored procedure must return data with a single result
+#set, that is the stored procedure must execute exactly one
+#SELECT statement on every code path. If you have complex
+#logic and for some paths you want to return no result you
+#will need to include a SELECT statement that returns no
+#rows. One example of a query that returns no rows is
+#
+# .nf
+#SELECT 1 FROM DUAL WHERE FALSE
+# .fi
+#
+#but you may use your own query.
+#
+#Stored procedures that return multiple result sets containing
+#data, that is stored procedures that execute multiple SELECT
+#statements, are not supported.  Stored procedures in
+#mysql produce an additional result set without data which
+#indicates the final status of the stored procedure. If this
+#final status is an error then any previous returned data
+#will not be used.
+#
+# The following is an example of a stored procedure returning
+# a single result set containing data:
+#
+# .nf
+#CREATE [DEFINER=`user`@`host`] PROCEDURE
+#`lookup`(IN `param` VARCHAR(255))
+#READS SQL DATA
+#SQL SECURITY INVOKER
+#BEGIN
+#select goto from alias where address=param;
+#END
+# .fi
 # OBSOLETE QUERY INTERFACE
 # .ad
 # .fi
diff -ur postfix-3.2-20161204_orig/src/global/dict_mysql.c
postfix-3.2-20161204/src/global/dict_mysql.c
--- postfix-3.2-20161204_orig/src/global/dict_mysql.c2016-09-25
17:27:11.0 +0200
+++ postfix-3.2-20161204/src/global/dict_mysql.c2016-12-24
19:56:24.967466042 +0100
@@ -187,6 +187,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #ifdef STRCASECMP_IN_STRINGS_H
 #include 
@@ -519,6 +520,10 @@
 {
 HOST   *host;
 MYSQL_RES *res = 0;
+MYSQL_RES *temp_res = 0;
+int status=0;
+int num_result_sets=0;
+int query_error=0;
 
 while ((host = dict_mysql_get_active(dict_mysql)) != NULL) {
 #if defined(MYSQL_VERSION_ID) && MYSQL_VERSION_ID >= 4
@@ -536,9 +541,56 @@
 #endif
 
 if (!(mysql_query(host->db, vstring_str(query {
-if ((res = mysql_store_result(host->db)) == 0) {
+query_error = 0;
+num_result_sets = 0;
+res = 0;
+do {
+temp_res = mysql_store_result(host->db);
+/* did statement 

Re: AW: Possible Bug ? postfix 3.1.0-3 fails on mysql table lookup

[Wietse Venema]

I have been working this code into Postfix, and have a comment
about error reporting for old-style queries.

>  while ((host = dict_mysql_get_active(dict_mysql)) != NULL) {
>  #if defined(MYSQL_VERSION_ID) && MYSQL_VERSION_ID >= 4
> @@ -536,9 +541,46 @@
>  #endif
> 
> if (!(mysql_query(host->db, vstring_str(query {
> -   if ((res = mysql_store_result(host->db)) == 0) {
> -   msg_warn("mysql query failed: %s", mysql_error(host->db));
> +   sp_error = 0;

Here, the old code reported "mysql query failed" when an old-style
query returned no result set.

> +   num_rs = 0;
> +   res = 0;
> +   do {
> +   temp_res = mysql_store_result(host->db);
> +   /* did statement return data? */
> +   if (temp_res) {
> +   num_rs++;
> +   if (num_rs > 1) {
> +   msg_warn("dict_mysql: multiple result sets
> returning data are not supported");
> +   mysql_free_result(temp_res);
> +   temp_res = 0;
> +   } else {
> +   res = temp_res;
> +   }
> +   } else {
> +   /* no data, check if there were errors */
> +   if (mysql_field_count(host->db) == 0) {
> +   if (num_rs == 0) {
> +   msg_warn("dict_mysql: stored procedure did
> not return any result set containing data");
> +   } else {
> +   if (msg_verbose)
> +   msg_info("dict_mysql: successful final
> result for stored procedure");
> +   }
> +   } else {
> +   sp_error=1;
> +   msg_warn("dict_mysql: unsuccessful final result
> for stored procedure: %s", mysql_error(host->db));

What if Postfix made an old-style query?  I think it should just
report the old-style error here.

Wietse

Re: AW: Possible Bug ? postfix 3.1.0-3 fails on mysql table lookup

[John Fawcett]

On 11/27/2016 01:47 PM, John Fawcett wrote:
> On 11/23/2016 10:54 PM, j...@conductive.de wrote:
>> On 2016-11-23 21:57, John Fawcett wrote:
>>> On 11/22/2016 01:35 AM, Joel Linn wrote:
 Hey Guys,

 this issue has decayed a bit but I now finally found the time (and the
 nerves) to integrate the fix in my system.
 I'm running Ubuntu 16.04 and trying not change to many things and be
 able to have clean comparison I applied the patch to the apt sources
 and only replaced the postfix-mysql package (ubuntu is using 3.1.0 it
 seems) with my own.
 I introduced some stored procedures and from my tests I conclude it
 works. I will see in the next couple of days if there are issues I
 overlooked.
 I ran into an issue where I was returning no result set using
 "smtpd_recipient_restrictions = check_recipient_access mysql:/[...]".
 I'm not sure if that's an dict_mysql issue or caused by design
 upstream. I overcame that by doing "select 1 from dual where false"
 when I don't have anything else to return, which basically is an empty
 result set.

 Thanks very much for your work,
 Joel
>>> Joel
>>>
>>> can you provide some more information to reproduce the issue you
>>> mentioned?
>>>
>>> John
>> Sure I can. Have a look at my log:
>>
>> Nov 23 22:42:49 leuchtkanone postfix/smtpd[5863]: connect from
>> divmail1.helmholtz-berlin.de[134.30.104.21]
>> Nov 23 22:42:49 leuchtkanone postfix/smtpd[5863]: warning:
>> mysql:/etc/postfix/sql/recipient-access.cf: table lookup problem
>> Nov 23 22:42:49 leuchtkanone postfix/smtpd[5863]: NOQUEUE: reject:
>> RCPT from divmail1.helmholtz-berlin.de[134.30.104.21]: 451 4.3.5
>> : Recipient address rejected: Server configuration
>> error; from= to=
>> proto=ESMTP helo=
>>
>> I used this empty test procedure (query = CALL ret_empty;):
>>
>> CREATE DEFINER=`vmail`@`localhost` PROCEDURE `ret_empty`()
>> BEGIN
>> END
>>
>> In my procedure I had some code paths return a result and some not.
>> Like I said i'm now using "SELECT 1 FROM dual WHERE false;" for those
>> dead paths, this simulates the behavior of the previous solution (one
>> line query) which always returns a result set, even if it's empty.
> Joel
>
> with MySQL procedures every SELECT statement that is executed produces a
> result set. In my patch I explicitly check to make sure there are no
> multiple result sets, but I do not check for no result. That could be
> improved upon, to give a specific warning that no result sets were
> returned. However, that will always be a lookup failure. The stored
> procedure (based on the way the patch is written) must always return a
> result set even if that result set is empty.
>
> Your solution with returning an empty result set when there is no other
> result (SELECT 1 FROM DUAL WHERE FALSE) looks correct.
>
> John
>
Revised patch to improve error reporting when no result set containing
data is returned

diff -ur postfix-3.2-20161106-orig/proto/mysql_table
postfix-3.2-20161106/proto/mysql_table
--- postfix-3.2-20161106-orig/proto/mysql_table 2016-10-01
15:44:52.0 +0200
+++ postfix-3.2-20161106/proto/mysql_table  2016-11-27
10:08:05.535236131 +0100
@@ -289,6 +289,39 @@
 #  certificate.
 # .sp
 #  This parameter is available with Postfix 2.11 and later.
+# USING MYSQL STORED PROCEDURES
+# .ad
+# .fi
+#  With Postfix 3.2 it is possible to call a stored procedure
+#  instead of using a SELECT statement in the query, e.g.
+#
+# .nf
+#  query = CALL lookup('%s')
+# .fi
+#
+#  The preivously described '%' expansions can be used in the
+#  parameter(s) to the stored procedure.
+#
+#  The stored procedure must return data with a single result
+#  set. Multiple result sets are not supported and will
+#  produce a warning and no valid result. Stored procedures in
+#  mysql produce an additional result set without data which
+#  indicates the final status of the stored procedure. If this
+#  final status is an error then any previous returned data
+#  will not be used.
+#
+#  The following is an example of a stored procedure returning
+#  data with a single result set:
+#
+# .nf
+#  CREATE [DEFINER=`user`@`host`] PROCEDURE
+#  `lookup`(IN `param` VARCHAR(255))
+#  READS SQL DATA
+#  SQL SECURITY INVOKER
+#  BEGIN
+#  select goto from alias where address=param;
+#  END
+# .fi
 # OBSOLETE QUERY INTERFACE
 # .ad
 # .fi
diff -ur postfix-3.2-20161106-orig/src/global/dict_mysql.c
postfix-3.2-20161106/src/global/dict_mysql.c
--- postfix-3.2-20161106-orig/src/global/dict_mysql.c   2016-09-25
17:27:11.0 +0200
+++ postfix-3.2-20161106/src/global/dict_mysql.c2016-11-27
13:56:50.956334035 +0100
@@ -187,6 +187,7 @@
 #include 
 #include 
 #include 
+#include 

 #ifdef STRCASECMP_IN_STRINGS_H
 #include 
@@ -519,6 +520,10 @@
 

Re: AW: Possible Bug ? postfix 3.1.0-3 fails on mysql table lookup

[John Fawcett]

On 11/23/2016 10:54 PM, j...@conductive.de wrote:
> On 2016-11-23 21:57, John Fawcett wrote:
>> On 11/22/2016 01:35 AM, Joel Linn wrote:
>>> Hey Guys,
>>>
>>> this issue has decayed a bit but I now finally found the time (and the
>>> nerves) to integrate the fix in my system.
>>> I'm running Ubuntu 16.04 and trying not change to many things and be
>>> able to have clean comparison I applied the patch to the apt sources
>>> and only replaced the postfix-mysql package (ubuntu is using 3.1.0 it
>>> seems) with my own.
>>> I introduced some stored procedures and from my tests I conclude it
>>> works. I will see in the next couple of days if there are issues I
>>> overlooked.
>>> I ran into an issue where I was returning no result set using
>>> "smtpd_recipient_restrictions = check_recipient_access mysql:/[...]".
>>> I'm not sure if that's an dict_mysql issue or caused by design
>>> upstream. I overcame that by doing "select 1 from dual where false"
>>> when I don't have anything else to return, which basically is an empty
>>> result set.
>>>
>>> Thanks very much for your work,
>>> Joel
>> Joel
>>
>> can you provide some more information to reproduce the issue you
>> mentioned?
>>
>> John
> Sure I can. Have a look at my log:
>
> Nov 23 22:42:49 leuchtkanone postfix/smtpd[5863]: connect from
> divmail1.helmholtz-berlin.de[134.30.104.21]
> Nov 23 22:42:49 leuchtkanone postfix/smtpd[5863]: warning:
> mysql:/etc/postfix/sql/recipient-access.cf: table lookup problem
> Nov 23 22:42:49 leuchtkanone postfix/smtpd[5863]: NOQUEUE: reject:
> RCPT from divmail1.helmholtz-berlin.de[134.30.104.21]: 451 4.3.5
> : Recipient address rejected: Server configuration
> error; from= to=
> proto=ESMTP helo=
>
> I used this empty test procedure (query = CALL ret_empty;):
>
> CREATE DEFINER=`vmail`@`localhost` PROCEDURE `ret_empty`()
> BEGIN
> END
>
> In my procedure I had some code paths return a result and some not.
> Like I said i'm now using "SELECT 1 FROM dual WHERE false;" for those
> dead paths, this simulates the behavior of the previous solution (one
> line query) which always returns a result set, even if it's empty.

Joel

with MySQL procedures every SELECT statement that is executed produces a
result set. In my patch I explicitly check to make sure there are no
multiple result sets, but I do not check for no result. That could be
improved upon, to give a specific warning that no result sets were
returned. However, that will always be a lookup failure. The stored
procedure (based on the way the patch is written) must always return a
result set even if that result set is empty.

Your solution with returning an empty result set when there is no other
result (SELECT 1 FROM DUAL WHERE FALSE) looks correct.

John

Re: AW: Possible Bug ? postfix 3.1.0-3 fails on mysql table lookup

[jl]

On 2016-11-23 21:57, John Fawcett wrote:

On 11/22/2016 01:35 AM, Joel Linn wrote:

Hey Guys,

this issue has decayed a bit but I now finally found the time (and the
nerves) to integrate the fix in my system.
I'm running Ubuntu 16.04 and trying not change to many things and be
able to have clean comparison I applied the patch to the apt sources
and only replaced the postfix-mysql package (ubuntu is using 3.1.0 it
seems) with my own.
I introduced some stored procedures and from my tests I conclude it
works. I will see in the next couple of days if there are issues I
overlooked.
I ran into an issue where I was returning no result set using
"smtpd_recipient_restrictions = check_recipient_access mysql:/[...]".
I'm not sure if that's an dict_mysql issue or caused by design
upstream. I overcame that by doing "select 1 from dual where false"
when I don't have anything else to return, which basically is an empty
result set.

Thanks very much for your work,
Joel

Joel

can you provide some more information to reproduce the issue you 
mentioned?


John

Sure I can. Have a look at my log:

Nov 23 22:42:49 leuchtkanone postfix/smtpd[5863]: connect from 
divmail1.helmholtz-berlin.de[134.30.104.21]
Nov 23 22:42:49 leuchtkanone postfix/smtpd[5863]: warning: 
mysql:/etc/postfix/sql/recipient-access.cf: table lookup problem
Nov 23 22:42:49 leuchtkanone postfix/smtpd[5863]: NOQUEUE: reject: RCPT 
from divmail1.helmholtz-berlin.de[134.30.104.21]: 451 4.3.5 
: Recipient address rejected: Server configuration 
error; from= to= 
proto=ESMTP helo=


I used this empty test procedure (query = CALL ret_empty;):

CREATE DEFINER=`vmail`@`localhost` PROCEDURE `ret_empty`()
BEGIN
END

In my procedure I had some code paths return a result and some not. Like 
I said i'm now using "SELECT 1 FROM dual WHERE false;" for those dead 
paths, this simulates the behavior of the previous solution (one line 
query) which always returns a result set, even if it's empty.

Re: AW: Possible Bug ? postfix 3.1.0-3 fails on mysql table lookup

[John Fawcett]

On 11/22/2016 01:35 AM, Joel Linn wrote:
> Hey Guys,
>
> this issue has decayed a bit but I now finally found the time (and the
> nerves) to integrate the fix in my system.
> I'm running Ubuntu 16.04 and trying not change to many things and be
> able to have clean comparison I applied the patch to the apt sources
> and only replaced the postfix-mysql package (ubuntu is using 3.1.0 it
> seems) with my own.
> I introduced some stored procedures and from my tests I conclude it
> works. I will see in the next couple of days if there are issues I
> overlooked.
> I ran into an issue where I was returning no result set using
> "smtpd_recipient_restrictions = check_recipient_access mysql:/[...]".
> I'm not sure if that's an dict_mysql issue or caused by design
> upstream. I overcame that by doing "select 1 from dual where false"
> when I don't have anything else to return, which basically is an empty
> result set.
>
> Thanks very much for your work,
> Joel
Joel

can you provide some more information to reproduce the issue you mentioned?

John

Re: AW: Possible Bug ? postfix 3.1.0-3 fails on mysql table lookup

[Joel Linn]

Hey Guys,

this issue has decayed a bit but I now finally found the time (and the 
nerves) to integrate the fix in my system.
I'm running Ubuntu 16.04 and trying not change to many things and be 
able to have clean comparison I applied the patch to the apt sources and 
only replaced the postfix-mysql package (ubuntu is using 3.1.0 it seems) 
with my own.
I introduced some stored procedures and from my tests I conclude it 
works. I will see in the next couple of days if there are issues I 
overlooked.
I ran into an issue where I was returning no result set using 
"smtpd_recipient_restrictions = check_recipient_access mysql:/[...]".
I'm not sure if that's an dict_mysql issue or caused by design upstream. 
I overcame that by doing "select 1 from dual where false" when I don't 
have anything else to return, which basically is an empty result set.


Thanks very much for your work,
Joel

On 2016-07-07 00:46, John Fawcett wrote:

On 07/04/2016 09:58 PM, j...@conductive.de wrote:


Quoting John Fawcett :
I can propose a code submission to add stored procedure support 
(based

on the proof of concept code from 2008), but the biggest part will be
doing the testing and non regression testing not the actual coding.

I believe the best approach to adding stored procedure support is to
iterate over multiple result sets and if there are exactly two and 
the

last one is the status result of the stored procedure and it is
successful then allow the lookup against the first result set. In all
other cases there should be an error for multiple result sets. In the
context of Postfix don't think it is useful to have more than one 
result

set generated by multiple queries or more than two result sets
(including the status one) from a stored procedure.

John


That is exactly what I had on my mind.

The specific error message would enable admins to exactly see where
the trouble originates.
Also consuming all results still allows subsequent lookups to succeed
normally.

I would of course volunteer to "test" the change per se but I'm afraid
my knowledge of postfix's inner workings converges to zero, rendering
my testing unrepresentative at last :/
Although the subset of postfix interfacing with mysql is quite small.

Joel


Hi

here is my proposed submission to add mysql stored procedure support to
Postfix. As per Wietse's comments in the following thread

http://postfix.1071664.n5.nabble.com/fatal-table-lookup-problem-with-Postfix-lookup-call-to-MySQL-StoredProcedure-td13240.html

"I would be grateful if that support came with test cases for single
and multi-result queries, including non-error and error results so
that we can run the test whenever something in the code is changed."

One further thing to point out and that may need to be adjusted still:

An error is returned whenever:

- the check for further result sets gives an error instead of yes (0) 
or

no (-1)

- the stored procedure final result set with the status is not as
expected (mysql_field_count returns 0)

- there is more than one result set containing data.

While testing with postmap and multiple result sets I was able to
generate the following error message from postmap.c

"postmap: fatal: table mysql:./mysql-test2.cf: query error: Success"

The final "Success" comes from %m parameter reporting the errno. Since
no system call error has been returned, but it is an application error 
I

have forced errno to ENOTSUP. If instead the "Success" status is deemed
correct, then the statement setting errno and the inclusion of errno.h
can be elimianted without changing the basic functioning of the patch.

I am available to make any further adjustments needed. I look forward 
to
feedback from Joel or other interested people about more extensive 
testing.


John

Re: AW: Possible Bug ? postfix 3.1.0-3 fails on mysql table lookup

[Wietse Venema]

John Fawcett:
> here is my proposed submission to add mysql stored procedure support to
> Postfix. As per Wietse's comments in the following thread

Thanks much. I'll examine it in the crumbs of available time.

Wietse