[pfx] Re: header_checks not work with regexp
On Fri, Apr 14, 2023 at 05:51:21PM -0300, SysAdmin EM via Postfix-users wrote: > postmap -q "Subject: [KIS] ERROR (EXTERNAL IP): Invalid HTTP_HOST header: > '10.54.130.188:8020'. You may need to add u'10.54.130.188' to ALLOWED_HOSTS." > regexp:/etc/postfix/header_checks Works here (bash syntax): $ hdr="Subject: [KIS] ERROR (EXTERNAL IP): Invalid HTTP_HOST header: '10.54.130.188:8020'. You may need to add u'10.54.130.188' to ALLOWED_HOSTS." $ rule='/^Subject:.*You may need to add.*/ DISCARD BLOCK_TEMPORAL' $ postmap -q "$hdr" regexp:<(printf "%s\n" "$rule") DISCARD BLOCK_TEMPORAL > any ideas?? Your testing methodology is flawed or regexp rule file is malformed. The file should contain: /^Subject:.*You may need to add.*/ DISCARD BLOCK_TEMPORAL on a single line with no leading whitespace, and not have any syntax issues on any other lines. -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: temporary lookup error with utf8mb4 characters
Viktor Dukhovni via Postfix-users: > On Fri, Apr 14, 2023 at 01:06:16PM -0400, Wietse Venema via Postfix-users > wrote: > > > Wietse Venema via Postfix-users: > > > As for the temp error becoming persistent, the Postfix pgsql: client > > > code returns an error when it gets an error from all of the hosts > > > configured in the Postfix pgsql: client configuration file, or when > > > all hosts have been flagged as 'down'. If a host returns an error > > > then the Postfix pgsql: client code flags that host as 'down', and > > > resets that 'down' state after about 60 seconds. > > > > As implemented, the Postfix pgsql: clien code treats all errors as > > a connection failure, and skips the connection for 60 seconds. That > > may not be optimal when an error is data dependent. > > FWIW, the OP's issue was with MySQL, not Postgres... The database > should be configured for client and server encoding of UTF8. Oops. Where did I get that from. There is no hard-coded encoding type in the Postfix mysql client. With smtputf8_enable=yes, Postfix will accept email addresses with well-formed UTF8 (and ASCII) but not Latin1. Wietse ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: header_checks not work with regexp
The rule not work postmap -q "Subject: [KIS] ERROR (EXTERNAL IP): Invalid HTTP_HOST header: ' 10.54.130.188:8020'. You may need to add u'10.54.130.188' to ALLOWED_HOSTS." regexp:/etc/postfix/header_checks WARN /^Subject:.*You may need to add.*/ DISCARD BLOCK_TEMPORAL any ideas?? On Fri, Apr 14, 2023 at 3:50 PM Viktor Dukhovni via Postfix-users < postfix-users@postfix.org> wrote: > On Fri, Apr 14, 2023 at 03:31:17PM -0300, SysAdmin EM via Postfix-users > wrote: > > > Sorry for such a basic question but I couldn’t find a solution on my > > part. I’m trying to block a Subject using header_checks but it’s not > > working. > > > > This is my rule: > > > > /^Subject:.*Invalid HTTP_HOST header.*/ DISCARD SUBJECTALERT > > This looks OK, and expects a "Subject:" header. > > > I used postmap to test the rule but not match. > > > > postmap -q "[KIS] ERROR (EXTERNAL IP): Invalid HTTP_HOST header: ' > 10.54.130.188:8020'. You may need to add u'10.54.130.188' to > ALLOWED_HOSTS." regexp:/etc/postfix/header_checks > > This is not a "Subject:" header. Perhaps you meant to type: > > postmap -q "Subject: [KIS] ERROR (EXTERNAL IP): Invalid HTTP_HOST > header: '10.54.130.188:8020'. You may need to add u'10.54.130.188' to > ALLOWED_HOSTS." regexp:/etc/postfix/header_checks > > -- > Viktor. > ___ > Postfix-users mailing list -- postfix-users@postfix.org > To unsubscribe send an email to postfix-users-le...@postfix.org > ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: temporary lookup error with utf8mb4 characters
Hi, I have more info and I try to explain it better:
First of all I have smtp_utf8 = no (disabled).
I have several databases related with several mysql_virtual maps:
- Some with utf8 + utf8_general_ci collation
- Another ones with latin1 + latin1_spanish_ci.
I'm using mysql-postfix (mysql_table) lookups, not postgres.
"proxy:mysql:/XXX.cf".
I can reproduce same issue with both cf files (tables with utf8 and
tables with latin1).
As I say before, the worst part is when error is raised during about 1
minute all lookups raises failures.
Error is easy to reproduce manually calling to "postmap -q
"emailWithspecialchar" "proxy:mysql:/XXX.cf"
Debugging I observe 2 things.
- adding CONVERT('%s' using ascii) fix the issue but I don't want/like
add converts on all my sql queries...
- adding COLLATE utf8_general_ci raises error "this collate is not valid
for utf8mb4". This error shows me than mysql_table lookup connections
are using "utf8mb4" charset by default.
My conclusion to hard-solve this issue on my system is transform all
tables to utf8mb4.
But:
- I don't see any option to change default charset on mysql_table
connector, maybe should be interesting add this option on configuration
file.
- mix collation error should raise 1 error, but next queries should be
work ok, this could be considered and issue right?.
- with "smtputf8_enable = no" I should be able to work without this kind
of issues right?
For modern protocols I can undestant change to utf8, but utf8mb4? this
is much more expensive for the database, is it really necessary?
**
El 14/4/23 a las 20:46, Viktor Dukhovni via Postfix-users escribió:
On Fri, Apr 14, 2023 at 01:06:16PM -0400, Wietse Venema via Postfix-users wrote:
Wietse Venema via Postfix-users:
As for the temp error becoming persistent, the Postfix pgsql: client
code returns an error when it gets an error from all of the hosts
configured in the Postfix pgsql: client configuration file, or when
all hosts have been flagged as 'down'. If a host returns an error
then the Postfix pgsql: client code flags that host as 'down', and
resets that 'down' state after about 60 seconds.
As implemented, the Postfix pgsql: clien code treats all errors as
a connection failure, and skips the connection for 60 seconds. That
may not be optimal when an error is data dependent.
FWIW, the OP's issue was with MySQL, not Postgres... The database
should be configured for client and server encoding of UTF8.
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: header_checks not work with regexp
On Fri, Apr 14, 2023 at 03:31:17PM -0300, SysAdmin EM via Postfix-users wrote: > Sorry for such a basic question but I couldn’t find a solution on my > part. I’m trying to block a Subject using header_checks but it’s not > working. > > This is my rule: > > /^Subject:.*Invalid HTTP_HOST header.*/ DISCARD SUBJECTALERT This looks OK, and expects a "Subject:" header. > I used postmap to test the rule but not match. > > postmap -q "[KIS] ERROR (EXTERNAL IP): Invalid HTTP_HOST header: > '10.54.130.188:8020'. You may need to add u'10.54.130.188' to ALLOWED_HOSTS." > regexp:/etc/postfix/header_checks This is not a "Subject:" header. Perhaps you meant to type: postmap -q "Subject: [KIS] ERROR (EXTERNAL IP): Invalid HTTP_HOST header: '10.54.130.188:8020'. You may need to add u'10.54.130.188' to ALLOWED_HOSTS." regexp:/etc/postfix/header_checks -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: temporary lookup error with utf8mb4 characters
On Fri, Apr 14, 2023 at 01:06:16PM -0400, Wietse Venema via Postfix-users wrote: > Wietse Venema via Postfix-users: > > As for the temp error becoming persistent, the Postfix pgsql: client > > code returns an error when it gets an error from all of the hosts > > configured in the Postfix pgsql: client configuration file, or when > > all hosts have been flagged as 'down'. If a host returns an error > > then the Postfix pgsql: client code flags that host as 'down', and > > resets that 'down' state after about 60 seconds. > > As implemented, the Postfix pgsql: clien code treats all errors as > a connection failure, and skips the connection for 60 seconds. That > may not be optimal when an error is data dependent. FWIW, the OP's issue was with MySQL, not Postgres... The database should be configured for client and server encoding of UTF8. -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] header_checks not work with regexp
Sorry for such a basic question but I couldn’t find a solution on my part. I’m trying to block a Subject using header_checks but it’s not working. This is my rule: /^Subject:.*Invalid HTTP_HOST header.*/ DISCARD SUBJECTALERT I used postmap to test the rule but not match. postmap -q "[KIS] ERROR (EXTERNAL IP): Invalid HTTP_HOST header: '10.54.130.188:8020'. You may need to add u'10.54.130.188' to ALLOWED_HOSTS." regexp:/etc/postfix/header_checks in the configuration see support for regexp: postconf -m btree cidr environ fail hash internal memcache nis proxy regexp sdbm static tcp texthash unix postconf mail_version mail_version = 2.9.6 Is there something wrong with my rule? ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: temporary lookup error with utf8mb4 characters
Wietse Venema via Postfix-users:
> As for the temp error becoming persistent, the Postfix pgsql: client
> code returns an error when it gets an error from all of the hosts
> configured in the Postfix pgsql: client configuration file, or when
> all hosts have been flagged as 'down'. If a host returns an error
> then the Postfix pgsql: client code flags that host as 'down', and
> resets that 'down' state after about 60 seconds.
As implemented, the Postfix pgsql: clien code treats all errors as
a connection failure, and skips the connection for 60 seconds. That
may not be optimal when an error is data dependent.
What error did Postfix log for a bad query? It would be helpful if
you could look in your logs for the exact error message.
The relevant code is below my signature.
Wietse
if ((res = PQexec(host->db, vstring_str(query))) != 0) {
...
switch ((status = PQresultStatus(res))) {
case PGRES_TUPLES_OK:
case PGRES_COMMAND_OK:
/* Success. */
if (msg_verbose)
msg_info("dict_pgsql: successful query from host %s",
host->hostname);
event_request_timer(dict_pgsql_event, (void *) host,
dict_pgsql->idle_interval);
return (res);
case PGRES_FATAL_ERROR:
msg_warn("pgsql query failed: fatal error from host %s: %s",
host->hostname, PQresultErrorMessage(res));
break;
case PGRES_BAD_RESPONSE:
msg_warn("pgsql query failed: protocol error, host %s",
host->hostname);
break;
default:
msg_warn("pgsql query failed: unknown code 0x%lx from host %s",
(unsigned long) status, host->hostname);
break;
}
} else {
...
msg_warn("pgsql query failed: fatal error from host %s: %s",
host->hostname, PQerrorMessage(host->db));
}
...
/*
* XXX An error occurred. Clean up memory and skip this connection.
*/
...
plpgsql_down_host(dict_pgsql, host);
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: temporary lookup error with utf8mb4 characters
V?ctor Rubiella Monfort via Postfix-users:
> Hi again,
>
> I realized than same error is raised when database is in utf8 if email
> contains utf8mb4 characters.
>
> Which is the convenient database collation for postfix? We can force
> postfix to accept only utf8 characters?.
With "smtputf8_enable = yes" in main.cf, Postfix will accept
well-formed UTF-8. However, before Postfix 3.8, the Postfix pgsql:
client code sets the client encoding to LATIN1.
The client encoding is configurable (default: UTF8) with Postfix
3.8 which is to be released soon (postfix-3.8.0-RC1 was published
a few days ago).
As for the temp error becoming persistent, the Postfix pgsql: client
code returns an error when it gets an error from all of the hosts
configured in the Postfix pgsql: client configuration file, or when
all hosts have been flagged as 'down'. If a host returns an error
then the Postfix pgsql: client code flags that host as 'down', and
resets that 'down' state after about 60 seconds.
In your case having multiple hosts configured in the Postfix pgsql:
client configuration would not help, because they would have the
same error.
Wietse
>
>
> El 13/4/23 a las 18:36, V?ctor Rubiella Monfort via Postfix-users escribi?:
> > When mysql_table lookup is executing nonascii characters and database
> > is in latin1, not only fails query, all sesion/connection is corrupted
> > and produces a lot of "temporary lookup table" errors until sesion is
> > recreated (about 1 minute later).
> >
> > Today some external ip was trying to deliver an email with special
> > character on one on my legacy servers (with latin1) and produces this
> > errors.
> >
> > I can understant that lookup fails for query with special characters,
> > but main issue was for all raised failures for other accounts and
> > lookups during 1-2 minutes. This is a knew issue?.
> >
> >
> > I deploy an workaround using "CONVERT('%s' using ascii)" until not
> > pass all database tables to utf8.
> >
> > The main problem debuging this issue was "proxy:mysql" , "proxy" was
> > hiding original collation error and only shows regular lookup errors
> > on postfix log, when user "postmap" to debug, I only see root cause
> > when execute without "proxy".
> >
> > postfix versions tested:
> >
> > postfix 3.5.17-0+deb11u1
> > postfix-mysql??? 3.5.17-0+deb11u1
> >
> > postfix 3.5.15-0+deb11u1
> > postfix-mysql??? 3.5.15-0+deb11u1
> >
> >
> >
> >
> > ___
> > Postfix-users mailing list -- postfix-users@postfix.org
> > To unsubscribe send an email to postfix-users-le...@postfix.org
> ___
> Postfix-users mailing list -- postfix-users@postfix.org
> To unsubscribe send an email to postfix-users-le...@postfix.org
>
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: temporary lookup error with utf8mb4 characters
Hi again,
I realized than same error is raised when database is in utf8 if email
contains utf8mb4 characters.
Which is the convenient database collation for postfix? We can force
postfix to accept only utf8 characters?.
El 13/4/23 a las 18:36, Víctor Rubiella Monfort via Postfix-users escribió:
When mysql_table lookup is executing nonascii characters and database
is in latin1, not only fails query, all sesion/connection is corrupted
and produces a lot of "temporary lookup table" errors until sesion is
recreated (about 1 minute later).
Today some external ip was trying to deliver an email with special
character on one on my legacy servers (with latin1) and produces this
errors.
I can understant that lookup fails for query with special characters,
but main issue was for all raised failures for other accounts and
lookups during 1-2 minutes. This is a knew issue?.
I deploy an workaround using "CONVERT('%s' using ascii)" until not
pass all database tables to utf8.
The main problem debuging this issue was "proxy:mysql" , "proxy" was
hiding original collation error and only shows regular lookup errors
on postfix log, when user "postmap" to debug, I only see root cause
when execute without "proxy".
postfix versions tested:
postfix 3.5.17-0+deb11u1
postfix-mysql 3.5.17-0+deb11u1
postfix 3.5.15-0+deb11u1
postfix-mysql 3.5.15-0+deb11u1
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
