Re: Dovecot with MySQL over SSL.

[John Fawcett via dovecot]

On 18/07/2019 23:24, Reio Remma via dovecot wrote:
> Hello!
>
> I'm attempting to get Dovecot working with MySQL user database on
> another machine. I can connect to the MySQL (5.7.26) instance with SSL
> enabled:
>
>  mysql -h db.mrst.ee --ssl-ca=/etc/dovecot/ca.pem
> --ssl-cert=/etc/dovecot/client-cert.pem
> --ssl-key=/etc/dovecot/client-key.pem --ssl-cipher=DHE-RSA-AES256-SHA
> -u vmail -p
>
> However if I use the same values in dovecot-sql.conf.ext, I get the
> following error:
>
> Jul 19 00:20:18 turin dovecot: master: Dovecot v2.3.7 (494d20bdc)
> starting up for imap, lmtp, sieve (core dumps disabled)
> Jul 19 00:20:18 turin dovecot: auth-worker(82996): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): SSL connection
> error: protocol version mismatch - waiting for 1 seconds before retry
> Jul 19 00:20:18 turin dovecot: auth-worker(82996): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): SSL connection
> error: protocol version mismatch - waiting for 1 seconds before retry
> Jul 19 00:20:19 turin dovecot: auth-worker(82996): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): Connections
> using insecure transport are prohibited while
> --require_secure_transport=ON. - waiting for 5 seconds before retry
> Jul 19 00:20:19 turin dovecot: auth-worker(82996): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): Connections
> using insecure transport are prohibited while
> --require_secure_transport=ON. - waiting for 5 seconds before retry
>
> Database connection string:
>
> connect = host=db.mrst.ee dbname=vmail user=vmail password=stuff \
>     ssl_ca=/etc/dovecot/ca.pem \
>     ssl_cert=/etc/dovecot/client-cert.pem \
>     ssl_key=/etc/dovecot/client-key.pem \
>     ssl_cipher=DHE-RSA-AES256-SHA
>
> If I leave the ssl_cipher unset, I get:
>
> Jul 19 00:23:41 turin dovecot: auth-worker(83069): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): SSL connection
> error: Failed to set ciphers to use - waiting for 1 seconds before retry
>
> Any ideas?
>
> Thanks!
> Reio

One difference between your testing manually with mysql client and the
same configuration in dovecot is the "ssl_verify_server_cert" parameter.
Dovecot is setting it if it is not specified. So to make the tests the
same you should either specify the --ssl_verify_server_cert parameter to
mysql or set it to no in the dovecot configuration.

John

Re: Dovecot/MSQL issue

[John Fawcett via dovecot]

On 01/07/2019 09:48, lorek via dovecot wrote:
> Actually, it seems I may have been wrong in initial assumption that
> the issue with the client was that it was being identified to mysql as
> coming from localhost when connecting via tcp.
> This is what syslog indicated as a reason for the failure but its not
> the whole picture.
>
> As John mentioned I am trying to have dovecot connect over TCP to
> mysql (not using the socket), and the issue looked like the cause was
> the identified by portion of mysql being read by either mysql
> incorrectly or the domain portion being overwritten on dovecot's end
> (I don't know about the internals enough to say for sure where).
>
> Just as due dilligence, I added credentials for a mysql user
> identified by localhost and removed the jail since the dovecot error
> was stating that it failed for connection by user@'localhost' (where
> there weren't credentials).
> After adding the credentials, I performed all the usual mysql tests
> before moving testing up to dovecot and still get an auth failure. The
> log seems to be a bit of a red herring or at the minimum doesn't show
> the whole picture.
>
> Replacing the connection string host with the socket (host=localhost)
> and everything works, and using an external IP that's not 127.0.0.1
> works as expected as well. (confirmed by standing up two isolated
> mysql and dovecot containers and setting auth up over the bridge).
>
> If the issue was caused by user@'localhost' creating the credentials
> should have resolved it, and it didn't. So something weird is going on.
> I've got the environment built up in a dockerfile I can provide if
> anyone wants to dig into what's causing it.
>
> In the meantime due to time constraints, I'll just be working with the
> socket file from now for hosts running most of the mail stack all in one.
>
> Best Regards,
> Lorek.

Lorek

If you have user@localhost as a user in mysql you will be able to
connect with either of these options:

- a tcp socket via an ip address that resolves to localhost

- a unix socket via hostname localhost.

If you have user@127.0.0.1 as a user in mysql you will be able to
connect only via:

- a tcp socket using ip address 127.0.0.1.

John

Re: Dovecot/MSQL issue

[John Fawcett via dovecot]

On 30/06/2019 13:36, Aki Tuomi via dovecot wrote:
>>
>> Hello, I'm trying to work through an issue that cropped up on a server I've 
>> been working on and haven't found a very good workaround.
>>
>> Dovecot is operating in a jailed environment.
>>
>> The configuration in dovecot-sql.conf.ext has been set appropriately with 
>> the host=127.0.0.1 (which works from a jailed environment) and when dovecot 
>> attempts to auth it appears to perform a reverse dns lookup and overwrites 
>> the host with the result localhost before using the msql credentials 
>> 'user'@'localhost' which then fails after timing out. 
>>
>> I'm currently running version 2.2.33.2 of dovecot.
>>
>> The workaround seems to be to have two credentials for the same user. 
>> 'user@'localhost' and 'user'@'127.0.0.1'; 
>> postfix operates in a jail as well and works around the jail issue in the 
>> same way but doesn't overwrite the specified host with a reverse dns lookup.
>>
>> Anyone have any suggestions on how to handle this issue better? or maybe 
>> there's a way to force different behavior which I haven't yet found in the 
>> documentation?
>>
>> Any help is much appreciated.
>>
>> Best Regards.
> Hi!
>
> The 127.0.0.1 issue is actually a MySQL issue, unfortunately. 
>
> " On Unix, MySQL programs treat the host name localhost specially, in a way 
> that is likely different from what you expect compared to other network-based 
> programs: the client connects using a Unix socket file. The --socket option 
> or the MYSQL_UNIX_PORT environment variable may be used to specify the socket 
> name. "
>
> So not really something we could do much about.
>
> Aki

Aki

it is possible to force the use of tcp socket instead of unix socket by
specifying an ip instead of the hostname localhost. As I understood it
Nathan is specifying 127.0.0.1 so that will connect with tcp socket.

John

Re: Error: quota-status: Client sent invalid recipient address: Invalid character in path

[John Fawcett via dovecot]

On 28/06/2019 02:01, Lefteris Tsintjelis via dovecot wrote:
> Any idea what this is?
>
> Jun 26 23:16:34
> quota-status(u...@example.com)<4975>: Error:
> quota-status: Client sent invalid recipient address: Invalid character
> in path
>
> I have this enabled in my config
>
> service quota-status {
>   client_limit = 1
>   executable = quota-status -p postfix
>   unix_listener /var/spool/postfix/private/dovquota {
> mode = 0666
> user = postfix
>   }
> }
>
> Lefteris

this is given by the address validation routines (for example see
lib-smtp/smtp-address.c as a starting point).

Maybe it would be easier to approach it by looking at the email address
that gave the error.

Is the email address compliant to smtp standards? Does it contain utf8
characters? Does the email address correspond to a mailbox of the same
name with a quota to be checked?

John

Re: last login user tracking

[John Fawcett via dovecot]

On 22/06/2019 16:59, Lefteris Tsintjelis via dovecot wrote:
> On 22/6/2019 16:07, John Fawcett via dovecot wrote:
>> You can easily view the value in a readable format with the
>> FROM_UNIXTIME mysql funciton
>>
>> https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#function_from-unixtime
>>
>> To get remote ip
>>
>> define the rip field in your table and add it to the primary key
>>
>> add %r to the last_login_key
>>
>> last_login_key = last-login/%u/%d/%r
>>
>> add $rip to the pattern and fields section in your
>> /etc/dovecot/dovecot-last-login.conf
>>
>> map {
>>     pattern = shared/last-login/$user/$domain/$rip
>>     table = last_login
>>     value_field = last_login
>>     value_type = uint
>>
>>     fields {
>>     username = $user
>>     domain = $domain
>>    rip = $rip
>>     }
>> }
>>
>> By adding the rip field to the primary key of the table, you can end up
>> with multiple rows per username each with different ips, but if you
>> don't do that, the rip field is never updated after the first insert.
> Does it work with IPv4 as well as IPv6? What is the optimal way to
> declare the rip field as?

yes IPv4 and IPv6. The value is in a string format. You can define a
VARCHAR(45) to hold it since no IPv4 or IPv6 address will be longer than
that.

John

Re: last login user tracking

[John Fawcett via dovecot]

On 21/06/2019 23:13, David Mehler via dovecot wrote:
> Hello,
>
> I'm trying to get last login user tracking going, using Dovecot and
> mysql. I'm using:
>
> https://docs.iredmail.org/track.user.last.login.html
>
> and it is working, I am seeing an entry and it is updated, the problem
> is the last login value does not mean anything to me. I was wondering
> if it's possible to have the value be of a last login date and time,
> and also a remote ip would be helpful so I can see from where users
> are logging in from.
>
> Thanks.
> Dave.

You can easily view the value in a readable format with the
FROM_UNIXTIME mysql funciton

https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#function_from-unixtime

To get remote ip

define the rip field in your table and add it to the primary key

add %r to the last_login_key

last_login_key = last-login/%u/%d/%r

add $rip to the pattern and fields section in your
/etc/dovecot/dovecot-last-login.conf

map {
    pattern = shared/last-login/$user/$domain/$rip
    table = last_login
    value_field = last_login
    value_type = uint

    fields {
    username = $user
    domain = $domain
   rip = $rip
    }
}

By adding the rip field to the primary key of the table, you can end up
with multiple rows per username each with different ips, but if you
don't do that, the rip field is never updated after the first insert.
John

Re: more generic approach as for userdb? (was: Dict issue with PostgreSQL for last_login plugin (duplicate key))

[John Fawcett via dovecot]

On 23/05/2019 07:49, Steffen Kaiser via dovecot wrote:
> On Wed, 22 May 2019, John Fawcett via dovecot wrote:
>
> > an update when insert fails seems to be a MySQL specific extension to
> > standard Sql. So I think that it's clear that support for PostgreSql and
> > Sqlite  needs to be implemented. The same issue likely exist in other
> > plugins too, for example expire.
>
> > My doubts are around the right solution to adopt. Initially I thought
> > that there was a PostgreSql syntax similar to MySQL which could be
> > easily added to the code, but closer inspection shows that the
> > PostgreSql syntax requires specification of either a constraint name or
> > the index column(s) for the primary/unique keys.
>
> You mean the "target" in ON CONFLICT target action, right?
> http://www.postgresqltutorial.com/postgresql-upsert/
>
Yes, whereas MySQL uses a generic syntax not requiring specific info, as
far as I am aware PostgreSql requires the target. I tried without and
got an error.

> > Constraint names are nowhere specified in the dictionary map syntax and
> > it's not possible either to identify with 100% certainty the primary key
> > column(s).
>
> One could dive into Postgres-specifics to get it, but there are other
> SQLs, too; the quota plugin advertises to use TRIGGERs to turn an
> INSERT into an UPDATE silently, which is no general approach either.
> https://wiki2.dovecot.org/Quota/Dict
>
> > 1) logic which always tries to update and falls back to insert if the
> > update fails (or viceversa) for all sql dictionaries.
>
> > 2) updates to the map syntax so that either the constraint name or
> > primary key columns can be specified.
>
> > Ideas are welcome.
>
> Maybe, one should drop the automatic at all and let the user specify
> the commands manually like with the userdb/passwd. Hence, the generic
> SQL preparation code is already present. There could/should/would be
> documented lots of "best practice" settings for various backends.
>
> In fact, this approach would better fit into the open and more
> "general" base idea Dovecot uses in other places, IMHO.
>
thanks for that suggestion, it would mean moving away from a syntax
where other dictionary types use a map statement and sql wouldn't.
> Kind regards,
>
> -- Steffen Kaiser

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 21/05/2019 15:45, mabi via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Monday, May 20, 2019 12:37 AM, John Fawcett via dovecot
>  wrote:
>
>> So looking into this with a postgresql databse to work with: the
>> above query does not work. You have to specify either the column name
>> or the constraint name that you expect to be violated in order for
>> the update to take place.
>>
>> With a map like this one you're using
>>
>> |map { pattern = shared/last-login/$user/$domain table = last_login
>> value_field = last_login value_type = uint fields { username = $user
>> domain = $domain } }|
>>
>> there's no field name that is obviously the primary key. I've
>> reworked the patch to use the postgres default primary key constraint
>> name (tablename_pkey).
>>
> So as you mention the new query you adapted which includes the primary
> key works, I tested it manually against PostgreSQL 10.5.
>>
>> The attached fix should work in that case, although I feel it's not
>> general enough.
>>
> Unfortunately my compiling skills are quite poor and I did not manage
> to patch and recompile Dovecot on OpenBSD.
>
> Do you think your patch will make it into the Dovecot code?
>
I feel confident that the patch works as the query has been manually
verified and the code change is not complex to validate.

The last_login plugin does not work at the moment with PostgreSql and
probably does not work with Sqlite, given that the only logic that tries
an update when insert fails seems to be a MySQL specific extension to
standard Sql. So I think that it's clear that support for PostgreSql and
Sqlite  needs to be implemented. The same issue likely exist in other
plugins too, for example expire.

My doubts are around the right solution to adopt. Initially I thought
that there was a PostgreSql syntax similar to MySQL which could be
easily added to the code, but closer inspection shows that the
PostgreSql syntax requires specification of either a constraint name or
the index column(s) for the primary/unique keys.

Constraint names are nowhere specified in the dictionary map syntax and
it's not possible either to identify with 100% certainty the primary key
column(s).

The solution I adopted in the latest version of the patch is to use the
default primary key constraint name derived from the table name, but
that won't help if people define custom constraint names. That may be an
unlikely scenario so the fix is certainly better than AS-IS. However it
is not perfect and added to that is the fact that the PostgreSql
extension is available only from 9.5.

I have no issues to submit the patch officially, as long as Dovecot
developers agree. However it may be worthwhile reflecting on a more
structural change

1) logic which always tries to update and falls back to insert if the
update fails (or viceversa) for all sql dictionaries.

2) updates to the map syntax so that either the constraint name or
primary key columns can be specified.

Ideas are welcome.

John

Re: Further issues on FTS engine

[John Fawcett via dovecot]

On 20/05/2019 08:27, Joan Moreau via dovecot wrote:
>
> Hi,
>
> Additionally to the long list of problem on the FTS previously
> discussed, here a new:
>
>
> WHen I reset the indexes, the indexer-worker seems paralelleizing the
> indexing (which is good), however, the number available in "ps aux |
> grep dove" shows that it does not move:
>
> dovecot 28549 0.0 0.0 8620 3920 ? S 06:20 0:00 dovecot/indexer [0
> clients, 3 requests]
> mailuse+ 28550 98.6 0.1 167412 86916 ? R 06:20 5:28
> dovecot/indexer-worker [j...@grosjo.net  - 800/37755]
>
>
> Looking further, if I put a tracer in teh backend, it treats the
> *same* message several times in parallel, and therefore does not move
> very fast on the global indexing of the box
>
> ANy clue ?
>
> THanks
>
>
Hi Joan

which version of dovecot are you using? I tried running ps aux | grep
dove on latest version but I don't see this behaviour.

Can you post some evidence of the same message being indexed serveral
times in parallel?

John

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 22:45, John Fawcett via dovecot wrote:
> On 19/05/2019 22:37, John Fawcett via dovecot wrote:
>> On 19/05/2019 20:31, mabi via dovecot wrote:
>>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Sunday, May 19, 2019 7:36 PM, John Fawcett via dovecot
>>>  wrote:
>>>>
>>>> Attached is a tentative patch. I've verified no regression for
>>>> mysql. There should be no regression for sqlite as the code path is
>>>> identical.
>>>>
>>>> Are you able to test for pgsql? As mentioned by Akie it will break
>>>> for PostgresSql < 9.5 but probably it was not working anyway due to
>>>> duplicate keys. Whether this is a wider problem depends on whether
>>>> the insert code is being used for other purposes too.
>>>>
>>>> If you or someone can verify it works on PostgresSql >= 9.5, then
>>>> the next step will be to make it conditional on the version.
>>>>
>>> Thank you very much John for your patch, that's fantastic. I am on
>>> OpenBSD 6.5 and will recompile dovecot from the ports by adding your
>>> patch to it, I hope that works and will let you know if I managed.
>>> If I understand correctly the relevant binary file I need to replace
>>> is the following right:
>>>
>>> /usr/local/lib/dovecot/dict/libdriver_pgsql.so
>>>
>>> or are there any others I also need to replace in order to test? I
>>> am planning to test live by just replacing the relevant file(s) so
>>> that I hopefully don't need to re-install the whole dovecot package.
>>
>> I'm not sure how the source compilation works on OpenBSD, when I do
>> it on linux and run "make install" it installs all relevant
>> binaries/libraries.
>>
>> I saw one issue with the fix though, it does not correctly pull out
>> the username field. I'm wondering if the query can be rewritten not
>> to mention the name of the field that fails the constraint
>>
>> John
>>
> so basically if this works just as well:
>
> INSERT INTO last_logins (last_login,username,domain) VALUES
> (1558273000,'u...@domain.tld <mailto:u...@domain.tld>','domain.tld')
> ON CONFLICT DO UPDATE SET last_login=1558273000,domain='domain.tld';
>
> then the fix can be altered to attached file which is more similar to
> the MYSQL syntax and does not require extra logic to get the username
> field.
>
> John
>
>
So looking into this with a postgresql databse to work with: the above
query does not work. You have to specify either the column name or the
constraint name that you expect to be violated in order for the update
to take place.

With a map like this one you're using

|map { pattern = shared/last-login/$user/$domain table = last_login
value_field = last_login value_type = uint fields { username = $user
domain = $domain } }|

there's no field name that is obviously the primary key. I've reworked
the patch to use the postgres default primary key constraint name
(tablename_pkey).

The attached fix should work in that case, although I feel it's not
general enough.

John

--- dict-sql-private.h.orig 2019-05-19 19:00:12.395887496 +0200
+++ dict-sql-private.h  2019-05-19 19:04:00.147601310 +0200
@@ -13,6 +13,7 @@
HASH_TABLE(const char *, struct sql_prepared_statement *) 
prep_stmt_hash;
 
bool has_on_duplicate_key:1;
+   bool has_on_conflict_do_update:1;
 };
 
 #endif
--- dict-sql.c.orig 2019-05-19 18:58:02.435194691 +0200
+++ dict-sql.c  2019-05-20 00:25:56.558251260 +0200
@@ -105,8 +105,10 @@
i_zero(_set);
sql_set.driver = driver->name;
sql_set.connect_string = dict->set->connect;
-   /* currently pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* mysql and sqlite don't support "ON CONFLICT DO UPDATE" */
dict->has_on_duplicate_key = strcmp(driver->name, "mysql") == 0;
+   dict->has_on_conflict_do_update = strcmp(driver->name, "pgsql") == 0;
 
if (sql_db_cache_new(dict_sql_db_cache, _set, >db, error_r) < 
0) {
pool_unref();
@@ -1108,12 +1110,15 @@
 
str_append_str(prefix, suffix);
str_append_c(prefix, ')');
-   if (!dict->has_on_duplicate_key) {
+   if (dict->has_on_duplicate_key ) {
+   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
+   } else if(dict->has_on_conflict_do_update)  {
+   str_printfa(prefix, " ON CONFLICT ON CONSTRAINT %s_pkey DO 
UPDATE SET ",fields[0].map->table);
+   } else {
*stmt_r = sql_dict_transaction_stmt_init(ctx, str_c(prefix), 
);
return 0;
}
 
-   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
for (i = 0; i < field_count; i++) {
const char *first_value_field =
t_strcut(fields[i].map->value_field, ',');

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 22:37, John Fawcett via dovecot wrote:
> On 19/05/2019 20:31, mabi via dovecot wrote:
>>
>> ‐‐‐ Original Message ‐‐‐
>> On Sunday, May 19, 2019 7:36 PM, John Fawcett via dovecot
>>  wrote:
>>>
>>> Attached is a tentative patch. I've verified no regression for
>>> mysql. There should be no regression for sqlite as the code path is
>>> identical.
>>>
>>> Are you able to test for pgsql? As mentioned by Akie it will break
>>> for PostgresSql < 9.5 but probably it was not working anyway due to
>>> duplicate keys. Whether this is a wider problem depends on whether
>>> the insert code is being used for other purposes too.
>>>
>>> If you or someone can verify it works on PostgresSql >= 9.5, then
>>> the next step will be to make it conditional on the version.
>>>
>> Thank you very much John for your patch, that's fantastic. I am on
>> OpenBSD 6.5 and will recompile dovecot from the ports by adding your
>> patch to it, I hope that works and will let you know if I managed. If
>> I understand correctly the relevant binary file I need to replace is
>> the following right:
>>
>> /usr/local/lib/dovecot/dict/libdriver_pgsql.so
>>
>> or are there any others I also need to replace in order to test? I am
>> planning to test live by just replacing the relevant file(s) so that
>> I hopefully don't need to re-install the whole dovecot package.
>
> I'm not sure how the source compilation works on OpenBSD, when I do it
> on linux and run "make install" it installs all relevant
> binaries/libraries.
>
> I saw one issue with the fix though, it does not correctly pull out
> the username field. I'm wondering if the query can be rewritten not to
> mention the name of the field that fails the constraint
>
> John
>
so basically if this works just as well:

INSERT INTO last_logins (last_login,username,domain) VALUES
(1558273000,'u...@domain.tld <mailto:u...@domain.tld>','domain.tld') ON
CONFLICT DO UPDATE SET last_login=1558273000,domain='domain.tld';

then the fix can be altered to attached file which is more similar to
the MYSQL syntax and does not require extra logic to get the username field.

John



--- dict-sql-private.h.orig 2019-05-19 19:00:12.395887496 +0200
+++ dict-sql-private.h  2019-05-19 19:04:00.147601310 +0200
@@ -13,6 +13,7 @@
HASH_TABLE(const char *, struct sql_prepared_statement *) 
prep_stmt_hash;
 
bool has_on_duplicate_key:1;
+   bool has_on_conflict_do_update:1;
 };
 
 #endif
--- dict-sql.c.orig 2019-05-19 18:58:02.435194691 +0200
+++ dict-sql.c  2019-05-19 19:17:52.613253822 +0200
@@ -105,8 +105,10 @@
i_zero(_set);
sql_set.driver = driver->name;
sql_set.connect_string = dict->set->connect;
-   /* currently pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* mysql and sqlite don't support "ON CONFLICT DO UPDATE" */
dict->has_on_duplicate_key = strcmp(driver->name, "mysql") == 0;
+   dict->has_on_conflict_do_update = strcmp(driver->name, "pgsql") == 0;
 
if (sql_db_cache_new(dict_sql_db_cache, _set, >db, error_r) < 
0) {
pool_unref();
@@ -1108,12 +1110,15 @@
 
str_append_str(prefix, suffix);
str_append_c(prefix, ')');
-   if (!dict->has_on_duplicate_key) {
+   if (dict->has_on_duplicate_key ) {
+   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
+   } else if(dict->has_on_conflict_do_update)  {
+   str_append(prefix, " ON CONFLICT DO UPDATE SET ");
+   } else {
*stmt_r = sql_dict_transaction_stmt_init(ctx, str_c(prefix), 
);
return 0;
}
 
-   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
for (i = 0; i < field_count; i++) {
const char *first_value_field =
t_strcut(fields[i].map->value_field, ',');

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 20:31, mabi via dovecot wrote:
>
> ‐‐‐ Original Message ‐‐‐
> On Sunday, May 19, 2019 7:36 PM, John Fawcett via dovecot
>  wrote:
>>
>> Attached is a tentative patch. I've verified no regression for mysql.
>> There should be no regression for sqlite as the code path is identical.
>>
>> Are you able to test for pgsql? As mentioned by Akie it will break
>> for PostgresSql < 9.5 but probably it was not working anyway due to
>> duplicate keys. Whether this is a wider problem depends on whether
>> the insert code is being used for other purposes too.
>>
>> If you or someone can verify it works on PostgresSql >= 9.5, then the
>> next step will be to make it conditional on the version.
>>
> Thank you very much John for your patch, that's fantastic. I am on
> OpenBSD 6.5 and will recompile dovecot from the ports by adding your
> patch to it, I hope that works and will let you know if I managed. If
> I understand correctly the relevant binary file I need to replace is
> the following right:
>
> /usr/local/lib/dovecot/dict/libdriver_pgsql.so
>
> or are there any others I also need to replace in order to test? I am
> planning to test live by just replacing the relevant file(s) so that I
> hopefully don't need to re-install the whole dovecot package.

I'm not sure how the source compilation works on OpenBSD, when I do it
on linux and run "make install" it installs all relevant binaries/libraries.

I saw one issue with the fix though, it does not correctly pull out the
username field. I'm wondering if the query can be rewritten not to
mention the name of the field that fails the constraint

John

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 17:08, mabi via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Sunday, May 19, 2019 4:44 PM, John Fawcett via dovecot
>  wrote:
>
>> I don't have PostgresSql, would you be able to verify if this syntax
>> would work: INSERT INTO last_logins (last_login,username,domain)
>> VALUES (1558273000,'u...@domain.tld
>> <mailto:u...@domain.tld>','domain.tld') ON CONFLICT(username) UPDATE
>> SET last_login=1558273000,domain='u...@domain.tld
>> <mailto:u...@domain.tld>' It's important to check that this updates
>> only the single row for that user and it puts the right data in that
>> row. If it doesn't work can you give the correct syntax?
>
> So you nearly yes ;-) The only parameter missing was "DO" keyword
> before the "UPDATE". So the correct query would be:
>
> INSERT INTO last_logins (last_login,username,domain) VALUES
> (1558273000,'u...@domain.tld <mailto:u...@domain.tld>','domain.tld')
> ON CONFLICT (username) DO UPDATE SET
> last_login=1558273000,domain='domain.tld';
>
> I also adapted the domain='domain.tld' at the end of the query, you
> had domain='u...@domain.tld <mailto:u...@domain.tld>' but this is just
> a "content" detail which does not matter.
>
> Hope that helps. Let me know if I can do any further testing.
>
Attached is a tentative patch. I've verified no regression for mysql.
There should be no regression for sqlite as the code path is identical.

Are you able to test for pgsql? As mentioned by Akie it will break for
PostgresSql < 9.5 but probably it was not working anyway due to
duplicate keys. Whether this is a wider problem depends on whether the
insert code is being used for other purposes too.

If you or someone can verify it works on PostgresSql >= 9.5, then the
next step will be to make it conditional on the version.

John

John

--- dict-sql-private.h.orig 2019-05-19 19:00:12.395887496 +0200
+++ dict-sql-private.h  2019-05-19 19:04:00.147601310 +0200
@@ -13,6 +13,7 @@
HASH_TABLE(const char *, struct sql_prepared_statement *) 
prep_stmt_hash;
 
bool has_on_duplicate_key:1;
+   bool has_on_conflict_do_update:1;
 };
 
 #endif
--- dict-sql.c.orig 2019-05-19 18:58:02.435194691 +0200
+++ dict-sql.c  2019-05-19 19:17:52.613253822 +0200
@@ -105,8 +105,10 @@
i_zero(_set);
sql_set.driver = driver->name;
sql_set.connect_string = dict->set->connect;
-   /* currently pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* mysql and sqlite don't support "ON CONFLICT DO UPDATE" */
dict->has_on_duplicate_key = strcmp(driver->name, "mysql") == 0;
+   dict->has_on_conflict_do_update = strcmp(driver->name, "pgsql") == 0;
 
if (sql_db_cache_new(dict_sql_db_cache, _set, >db, error_r) < 
0) {
pool_unref();
@@ -1108,12 +1110,15 @@
 
str_append_str(prefix, suffix);
str_append_c(prefix, ')');
-   if (!dict->has_on_duplicate_key) {
+   if (dict->has_on_duplicate_key ) {
+   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
+   } else if(dict->has_on_conflict_do_update)  {
+   str_printfa(prefix, " ON CONFLICT (%s) DO UPDATE SET 
",fields[0].map->username_field);
+   } else {
*stmt_r = sql_dict_transaction_stmt_init(ctx, str_c(prefix), 
);
return 0;
}
 
-   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
for (i = 0; i < field_count; i++) {
const char *first_value_field =
t_strcut(fields[i].map->value_field, ',');

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 16:25, John Fawcett via dovecot wrote:
> INSERT INTO last_logins (last_login,username,domain) VALUES 
> (1558273000,'u...@domain.tld','domain.tld')

I don't have PostgresSql, would you be able to verify if this syntax would work:

INSERT INTO last_logins (last_login,username,domain) VALUES 
(1558273000,'u...@domain.tld','domain.tld') ONCONFLICT(username) UPDATE SET 
last_login=1558273000,domain='u...@domain.tld'

It's important to check that this updates only the single row for that user and 
it puts the right data in that row. If it doesn't work can you give the correct 
syntax?

John 

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 15:42, mabi via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Sunday, May 19, 2019 2:58 PM, John Fawcett via dovecot 
>  wrote:
>
>> I guess it is not supported for PostgreSQL or Sqlite, since they don't
>> have "ON DUPLICATE KEY UPDATE" statement which is what is being used in
>> MySQL.
> That's it, PostgreSQL does not have "ON DUPLICATE KEY" but instead uses "ON 
> CONFLICT" as documented here:
>
> https://www.postgresql.org/docs/10/sql-insert.html#SQL-ON-CONFLICT
>
>> You could verify the query being used by turning on query logging in
>> PostgresSQL.
> I enabled query logging as suggested and found out that Dovecot dict is not 
> using the "ON CONFLICT" feature of INSERT with PostgreSQL, as you can see 
> from the query below:
>
> INSERT INTO last_logins (last_login,username,domain) VALUES 
> (1558273000,'u...@domain.tld','domain.tld')
>
> For me this makes Dovecot's dictionary feature useless with PostgreSQL. 
> Should I open a bug for that? or is it more of a "feature request"?
>
> Regards,
> Mabi

You may be able to find a workaround, by redefining the table without a
primary key and then select the max(last_login) from the table, with
some periodic job that clears out the old entries.

Or you could make a workaround with the rules syntax from PostgresSql.

https://www.postgresql.org/docs/9.2/sql-createrule.html

Though probably the best thing is a patch to dovecot in order to support
last_login for PostgresSql.

John

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 12:52, mabi via dovecot wrote:
> Hello,
>
> I am trying to setup the last_login plugin on Dovecot 2.3 in order to log a 
> user's last IMAP login into a separate PostgreSQL table called last_login. I 
> am actually following that guide: 
> https://docs.iredmail.org/track.user.last.login.html with the exception that 
> I use PostgreSQL instead of MySQL.
>
> So far it works at least for the very first login of a user but it looks like 
> Dict always want to add a new record into my last_login table instead of 
> updating the existing record for a specific user. Here is the error message 
> from Dovecot:
>
> May 19 12:45:14-mbox1 dovecot: 
> imap(u...@domain.tld)<51915>: Error: last_login_dict: 
> Failed to write value: dict-server returned failure: sql dict: commit failed: 
> ERROR:  duplicate key value violates unique constraint "last_logins_pkey"
>
> It looks like Dict with PostgreSQL does not support UPDATEing an already 
> existing record in the table. Or am I missing something? or it does not work 
> with PostgreSQL? Which does not make much sense to me...
>
> Regards,
> Mabi
>
I guess it is not supported for PostgreSQL or Sqlite, since they don't
have "ON DUPLICATE KEY UPDATE" statement which is what is being used in
MySQL.

You could verify the query being used by turning on query logging in
PostgresSQL.

John

Re: Auto rebuilding of Solr indexes on settings change?

[John Fawcett via dovecot]

On 25/04/2019 13:32, Peter Mogensen via dovecot wrote:
> Hi,
>
> Looking at the source, it doesn't seem like fts-solr checks for settings
> changes using fts_index_have_compatible_settings() like fts-lucene does.
>
> Is there any special reason for why fts-solr shouldn't also rebuild
> indexes if settings has changed?
>
> /Peter

Peter

I guess that there is no setting for the fts-solr plugin that would be
require rebuilding to take place. If you change any fts-solr settings
(except for the url) I would expect the result in solr, providing there
were no errors, will be the same with or without a rebuild.

For fts-lucene there are settings which would change what is stored in
lucene. The relevant settings which are taken into account for lucene
index rebuild are in  fts_lucene_settings_checksum(): default_language,
whitespace_chars, normalize and no_snowball.

John

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 16:34, Larry Rosenman via dovecot wrote:
> yes, either with or without the domain.
>
>
> On Mon, Apr 15, 2019 at 9:33 AM John Fawcett via dovecot
> mailto:dovecot@dovecot.org>> wrote:
>
> On 15/04/2019 16:05, Larry Rosenman via dovecot wrote:
>> I have normal system users, some and some virtual domains (2
>> different flavors).
>>
>> I have this job that runs from cron on my own userid to archive mail
>> I want Dovecot to use my system password for mail authentication.
>> I, therefore, use PAM to authenticate system users
>> Everything works great, modulo getting the auto-index to be
>> visible using RoundCube (which if I log into roundcube using
>> @ it does). 
>> by default if I'm logged in as my normal OS user, default doveadm
>> commands (as issued from the shell or from my archive script) use
>> the user I'm logged in as *WITHOUT A DOMAIN*.
>>
>> I want to default PAM auth'd users to append @lerctr.org
>> <http://lerctr.org> (DOMAIN) to the OS user.
>>
> when they log in to roundcube as the OS user, right?
>
> John
>
>
>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> <mailto:larry...@gmail.com>
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

So if all works fine when you specify user@domain in roundcube, but you
want to be able to specify only username in roundcube and have roundcube
login as user@domain, then one solution would be to configure roundcube
to add a default domain to bare usernames. As far as I remember (it's a
while since I used roundcube) it is one of the configuration options.

John

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 16:05, Larry Rosenman via dovecot wrote:
> I have normal system users, some and some virtual domains (2 different
> flavors).
>
> I have this job that runs from cron on my own userid to archive mail
> I want Dovecot to use my system password for mail authentication.
> I, therefore, use PAM to authenticate system users
> Everything works great, modulo getting the auto-index to be visible
> using RoundCube (which if I log into roundcube using @
> it does). 
> by default if I'm logged in as my normal OS user, default doveadm
> commands (as issued from the shell or from my archive script) use the
> user I'm logged in as *WITHOUT A DOMAIN*.
>
> I want to default PAM auth'd users to append @lerctr.org
>  (DOMAIN) to the OS user.
>
when they log in to roundcube as the OS user, right?

John

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 13:43, Larry Rosenman via dovecot wrote:
> Again, this doesn't help with doveadm running as the local user, and
> also doesn't help with
> the PAM authentication.
>
> passdb {
>   driver = pam
>   #[session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=20]
>   #[cache_key=] []
>   args = failure_show_msg=yes session=yes max_requests=20
>   skip = authenticated
> }
>
> How can I default the domain for PAM authentication?
> (I've set auth_default_realm and it doesn't help in this case).

Larry

I guess I don't understand enough about your setup or what is not now
working.

My understanding was that everything is now working when logging in as
user@domain, but that you would like to login as user and have dovecot
treat that as though you had logged in as user@domain, but at this point
I admit I may have misinterpreted your emails.

What's also not clear for me is the purpose in your setup of the three
passdb methods (sql, static, and pam) and two userdb methods (sql and
passwd-file). That's why I've pointed you to the docs and I'm hestitant
to give specific advice that may leave you worse off. Others on the list
may have more insights.

John

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 12:59, Larry Rosenman via dovecot wrote:
> forgot to reply all.
>
> -- Forwarded message -
> From: *Larry Rosenman* mailto:larry...@gmail.com>>
> Date: Mon, Apr 15, 2019 at 5:58 AM
> Subject: Re: SOLR/Index?
> To: John Fawcett mailto:j...@voipsupport.it>>
>
>
> the local users (myself, my wife, a friend) can authenticate EITHER as
>  or @lerctr.org .
>
> switching to all virtual users is NOT going to happen. 
>
> If I login to roundcube with @lerctr.org  it
> finds the autoindexed mail.
>
> So, if I make everyone always authenticate as @lerctr.org
>  we should be fine.
>
> and change my script to do doveadm -u @ instead of
> depending on the local user running the cron job. 
>
> question: Is there some way to have dovecot change what it sees to be
> @lerctr.org  when they login as ?
>
>
Dovecot is very configurable, but it can also take some time, effort and
testing to get the configuration you want. Personally I don't mix user
types since it takes out an element of complexity.

For your case you might find it useful to look into auth_default_realm

that can specify a domain name when one is not supplied.

https://wiki2.dovecot.org/DomainLost

Other things that might be useful: there is a method for returning a
"user" field from the userdb query or passdb query which will change the
username. Or there is another setting that can overriding values of
fields returned by the userdb.

https://wiki.dovecot.org/UserDatabase

Hope it helps!

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:
> ⌂63% [l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> ⌂67% [l...@thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970"
> l...@lerctr.org 
> test fts1970
>
> test fts1970
> .
> EOT
> [l...@thebighonker.lerctr.org:~] $ mailq
> [l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> Subject: test fts1970
> test fts1970
> test fts1970
>
>
> Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <=
> l...@lerctr.org  U=ler P=local S=388
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from
> local
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364
> ): save: box=INBOX, uid=175402,
> msgid= >, size=640,
> vsize=660, from=Larry Rosenman  >, subject=test fts1970, flags=()
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364
> ): sieve:
> msgid= >: stored mail into
> mailbox 'INBOX' (subject=test fts1970 from=l...@lerctr.org
>  size=660)
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect
> from local: Client has quit the connection (state=READY)
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler
> mailto:l...@lerctr.org>> R=localuser T=dovecot_lmtp
> S=404 C="250 2.0.0 mailto:l...@lerctr.org>>
> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s
> Apr 15 04:29:03 thebighonker dovecot[2507]:
> indexer-worker(l...@lerctr.org/49366 ):
> Indexed 1 messages in INBOX (UIDs 175402..175402)
>
>
> ⌂81% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX 
> body 'fts1970'
> ⌂83% [l...@thebighonker.lerctr.org:~] $
>
>
> ⌂65% [l...@thebighonker.lerctr.org:~] 75 $ doveadm search -u
> l...@lerctr.org   mailbox INBOX body 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
> [l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $ doveadm search -u l...@lerctr.org
>   mailbox INBOX body 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $
>
> So, yes, your hypothesis is correct.
>
> Question: How can I make it consistent?  
>
> I have a script that runs on the first of the month that does
> archiving, and I have similar issues in that namespace:
> ⌂67% [l...@thebighonker.lerctr.org:~] $ cat bin/archive-mail
> #!/bin/sh
> PATH=$PATH:/usr/local/bin
> #Expects to be run after midnight on the first of the month
> #  to archive all the previous months mail
> #Date Run:
> TODAY=`date "+%Y-%m-%d"`
> #last month in /MM
> YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"`
> #1st of last month as 01-Mon-
> FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"`
> echo 'TODAY=' ${TODAY}
> echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH}
> echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH}
> # get a list of all the mailboxes with at least one real message
> doveadm -f tab mailbox status vsize \* 2>/dev/null |
>         sed -e 1d | sort -k 1,1 |
>         awk  'BEGIN {FS="\t"} {if ($2 > 0)  print $1}' |
> while read i
> do
>    echo `date` start ${i}
>    doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}"
>    doveadm -f tab mailbox status messages "${i}"
>    doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \
>             "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH}
>    doveadm -f tab mailbox status messages "${i}"
>    echo `date` done  ${i}
> done
> ⌂64% [l...@thebighonker.lerctr.org:~] $
>
>
> The Exim config can be provided as well if necessary.
>
> ler & l...@lerctr.org  *ARE THE SAME MAILBOX*
>
At the moment it looks as though you have two sets of emails indexed in
solr. One is indexed under username (the one you are running mannually
and apparently the one used by roundcube too, but that's to be verified)
and another set being indexed by autoindex = yes option using the full
email address. Once you've got it working as you require, then you may
want to clean out solr and reindex with just one of them just to reduce
volumes.

Your setup seems to have a mix of users from mysql and from /etc/passwd.
Not sure if your mysql users are all mapped to real users or they have
their own mailboxes with domain included. Your solution will depend on
what you really need and if the setup is working correctly you may not
want to tweak it too much or other things may 

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 10:59, Larry Rosenman via dovecot wrote:
> I'll run a full test when I'm back in front of areal computer vs. My
> phone.(in a few hours)
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> 
> *From:* dovecot  on behalf of John
> Fawcett via dovecot 
> *Sent:* Monday, April 15, 2019 3:57:08 AM
> *To:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>  
> On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:
>> It always shows the autoindex. And yes built from sources.  I'm the
>> FreeBSD port maintainer for mail/docecot.  This has been happening
>> for several releases.
>>
>> Get Outlook for Android <https://aka.ms/ghei36>
>>
>> ------------
>> *From:* dovecot  on behalf of John
>> Fawcett via dovecot 
>> *Sent:* Monday, April 15, 2019 2:06:55 AM
>> *To:* dovecot@dovecot.org
>> *Subject:* Re: SOLR/Index?
>>  
>> On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
>>> Note the hits after the fts rescan/index.
>>>
>>> Get Outlook for Android <https://aka.ms/ghei36>
>>>
>>> 
>>> *From:* Aki Tuomi 
>>> *Sent:* Monday, April 15, 2019 12:55:07 AM
>>> *To:* Larry Rosenman; John Fawcett
>>> *Cc:* Dovecot Mailing List
>>> *Subject:* Re: SOLR/Index?
>>>  
>>>
>>>
>>> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
>>>> ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
>>>> lists/freebsd/ports-commiters  body 'sysutils'
>>>> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
>>>> [l...@thebighonker.lerctr.org:~] $ doveadm index -q
>>>> lists/freebsd/ports-commiters
>>>> ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
>>>> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login:
>>>> Disconnected (auth failed, 1 attempts in 2 secs): user=,
>>>> method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS:
>>>> Connection closed, session=
>>>> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
>>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
>>>> session=
>>>> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
>>>> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>>> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
>>>> address 23.100.68.192
>>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>>> H=(DaVinci-MWare.prophet21lab.com
>>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130
>>>> I=[192.147.25.65]:25 sender verify defer for >>> <mailto:i...@duke.org>>: host lookup did not complete
>>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>>> H=(DaVinci-MWare.prophet21lab.com
>>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130
>>>> I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
>>>> temporarily rejected RCPT mailto:jpo...@why.net>>:
>>>> Could not complete sender verify
>>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
>>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
>>>> session=
>>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
>>>> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>>> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
>>>> address 80.253.235.35
>>>> Apr 14 19:31:19 thebighonker dovecot[2507]:
>>>> indexer-worker(ler/14919): Indexed 1578 messages in
>>>> lists/freebsd/ports-commiters (UIDs 21067..22644)
>>>> ^C
>>>> [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
>>>> lists/freebsd/ports-commiters  body 'sysutils/'
>>>
>>>
>>> Just minor nit, but you are searching for 'sysutils' first, then
>>> 'sysutils/'. FTS does not do substring searches by default.
>>>
>>> Aki
>>>
>>>>
>>>> -- 
>>>> Larry Rosenman                     http://www.lerctr.org/~ler
>>>> <http://www.lerctr.org/~ler>
>>>>

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:
> It always shows the autoindex. And yes built from sources.  I'm the
> FreeBSD port maintainer for mail/docecot.  This has been happening for
> several releases.
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> 
> *From:* dovecot  on behalf of John
> Fawcett via dovecot 
> *Sent:* Monday, April 15, 2019 2:06:55 AM
> *To:* dovecot@dovecot.org
> *Subject:* Re: SOLR/Index?
>  
> On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
>> Note the hits after the fts rescan/index.
>>
>> Get Outlook for Android <https://aka.ms/ghei36>
>>
>> 
>> *From:* Aki Tuomi 
>> *Sent:* Monday, April 15, 2019 12:55:07 AM
>> *To:* Larry Rosenman; John Fawcett
>> *Cc:* Dovecot Mailing List
>> *Subject:* Re: SOLR/Index?
>>  
>>
>>
>> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
>>> ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
>>> lists/freebsd/ports-commiters  body 'sysutils'
>>> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
>>> [l...@thebighonker.lerctr.org:~] $ doveadm index -q
>>> lists/freebsd/ports-commiters
>>> ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
>>> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected
>>> (auth failed, 1 attempts in 2 secs): user=, method=PLAIN,
>>> rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed,
>>> session=
>>> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
>>> session=
>>> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
>>> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
>>> address 23.100.68.192
>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>> H=(DaVinci-MWare.prophet21lab.com
>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130
>>> I=[192.147.25.65]:25 sender verify defer for >> <mailto:i...@duke.org>>: host lookup did not complete
>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>> H=(DaVinci-MWare.prophet21lab.com
>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130
>>> I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
>>> temporarily rejected RCPT mailto:jpo...@why.net>>:
>>> Could not complete sender verify
>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
>>> session=
>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
>>> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
>>> address 80.253.235.35
>>> Apr 14 19:31:19 thebighonker dovecot[2507]:
>>> indexer-worker(ler/14919): Indexed 1578 messages in
>>> lists/freebsd/ports-commiters (UIDs 21067..22644)
>>> ^C
>>> [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
>>> lists/freebsd/ports-commiters  body 'sysutils/'
>>
>>
>> Just minor nit, but you are searching for 'sysutils' first, then
>> 'sysutils/'. FTS does not do substring searches by default.
>>
>> Aki
>>
>>>
>>> -- 
>>> Larry Rosenman                     http://www.lerctr.org/~ler
>>> <http://www.lerctr.org/~ler>
>>> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
>>> <mailto:larry...@gmail.com>
>>> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>
> Larry
>
> just to be sure: are you running a standard unmodified 2.3.5.1 version
> which you built from source code?
>
> I can see that first you search for sysutils, then do a rescan and
> reindex (which is shown in the log) and then you are able to find
> sysutils/.
>
> It is better when doing these tests to search for the same string
> before and after, just to eliminate too many different factors in the
> test.
>
> Nevertheless I did not see your logging for what happens when you
> receive a test message containing sysutils/. Dovecot should be
> outputing info about autoindexing given your setup. Does it do that or
> does it give some other message? Can you show those logs?
>
> John
>
Larry

Did you notice any difference between the logging for auto indexing and
the logging for indexing that you triggered manually? Would you mind
posting the auto indexing logging for a message to that same user (ler)?

best regards

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
> Note the hits after the fts rescan/index.
>
> Get Outlook for Android 
>
> 
> *From:* Aki Tuomi 
> *Sent:* Monday, April 15, 2019 12:55:07 AM
> *To:* Larry Rosenman; John Fawcett
> *Cc:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>  
>
>
> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
>> ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
>> lists/freebsd/ports-commiters  body 'sysutils'
>> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
>> [l...@thebighonker.lerctr.org:~] $ doveadm index -q
>> lists/freebsd/ports-commiters
>> ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
>> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected
>> (auth failed, 1 attempts in 2 secs): user=, method=PLAIN,
>> rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed,
>> session=
>> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
>> session=
>> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
>> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
>> address 23.100.68.192
>> Apr 14 19:30:55 thebighonker exim[14846]:
>> H=(DaVinci-MWare.prophet21lab.com
>> ) [23.100.68.192]:52130
>> I=[192.147.25.65]:25 sender verify defer for > >: host lookup did not complete
>> Apr 14 19:30:55 thebighonker exim[14846]:
>> H=(DaVinci-MWare.prophet21lab.com
>> ) [23.100.68.192]:52130
>> I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
>> temporarily rejected RCPT mailto:jpo...@why.net>>:
>> Could not complete sender verify
>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
>> session=
>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
>> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
>> address 80.253.235.35
>> Apr 14 19:31:19 thebighonker dovecot[2507]:
>> indexer-worker(ler/14919): Indexed 1578 messages in
>> lists/freebsd/ports-commiters (UIDs 21067..22644)
>> ^C
>> [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
>> lists/freebsd/ports-commiters  body 'sysutils/'
>
>
> Just minor nit, but you are searching for 'sysutils' first, then
> 'sysutils/'. FTS does not do substring searches by default.
>
> Aki
>
>>
>> -- 
>> Larry Rosenman                     http://www.lerctr.org/~ler
>> 
>> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
>> 
>> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version
which you built from source code?

I can see that first you search for sysutils, then do a rescan and
reindex (which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before
and after, just to eliminate too many different factors in the test.

Nevertheless I did not see your logging for what happens when you
receive a test message containing sysutils/. Dovecot should be outputing
info about autoindexing given your setup. Does it do that or does it
give some other message? Can you show those logs?

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 01:39, Larry Rosenman via dovecot wrote:
>
> full solr.log at:
> https://www.lerctr.org/~ler/solr.log
>
> The search DOES make it to SOLR:
> ⌂77% [l...@thebighonker.lerctr.org:~] 130 $ grep sysutils
> /var/log/solr/solr.log
> 2019-04-14 18:31:34.749 INFO  (qtp349420578-7538) [   x:dovecot]
> o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select
> params={q={!lucene+q.op%3DAND}(hdr:sysutils\/+OR+body:sysutils\/)=uid,score=uid+asc=%2Bbox:8097632f69627b5b5895bbe98eac+%2Buser:ler=22644=xml}
> hits=0 status=0 QTime=460
>
> Pick showing subjects:
> https://www.lerctr.org/~ler/sysutils_mail.png
>
> What else?
>
> I'm happy to provide access.
>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> 
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

so the search is returning no hits as you said. But can you show that
there is data in the index that should match?

doveadm search -u u...@example.com mailbox inbox body "sysutils/"

Can you do a controlled test and send yourself a test message with that
string and show the solr log where it is being inserted into the index
and then search for it with doveadm (just to rule out roundcube for the
moment) and show solr log for that search?

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 01:15, Larry Rosenman via dovecot wrote:
> Given all the discussion on FTS/Solr, etc, I have a question:
>
> I have autoindex set, and searching in roundcube most of the time does
> NOT find things,
> HOWEVER if I do:
> doveadm fts rescan
> doveadm index
>
> I can find things in the mailboxes.
>
> WHY?
>
> (doveconf -n attached).
>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> 
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

have you been able to check your solr logs to see if the queries that
find nothing actually arrive at solr? What is logged for those queries?

Some clients (don't know if roundcube is among those) don't send the
query in some circumstances.

Also when email arrives in dovecot is the automatic indexing working: do
you see solr logging for adding those messages.

John

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 17:55, John Fawcett via dovecot wrote:
> The solr server is a small test virtual machine with 0.2 (shared) vCPU
> and 0.6MB of memory and non SSD storage. It can index around 2000 emails
> per minute when there is no other activity. Average email size is about
> 45Kb. I'm not indexing attachments.
>
> John
>
I more than double the performance by using a 0.5 shared vCPU and SSD
storage. Approx 4500 emails per minute.

John

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 17:16, Peter Mogensen via dovecot wrote:
> sorry... I got distracted half way and forgot to put a meaningfull
> subject so the archive could figure out the thread. - resending.
>
> On 4/14/19 4:04 PM, dovecot-requ...@dovecot.org wrote:
>
>>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>>> false on the autoCommit.? The autoSoftCommit setting is not enabled by
>>> default, but depending on how the index was created, Solr might try to
>>> set autoSoftCommit to 3 seconds ... which is WAY too short.
> I just run with the default. 15s autoCommit and no autoSoftCommit
>
>>> This thread says that dovecot is sending explicit commits.? 
> I see explicit /update req. with softCommit and waitSearcer=true in a
> tcpdump.
>
>>> One thing
>>> that might be happening to exceed 60 seconds is an extremely long
>>> commit, which is usually caused by excessive cache autowarming, but
>>> might be related to insufficient memory.? The max heap setting on an
>>> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY
>>> small, and it doesn't take much index data before a much larger heap
>>> is required.
> I run with
>
> SOLR_JAVA_MEM="-Xmx8g -Xms2g"
>
>> I looked into the code (version 2.3.5.1):
> This is 2.2.35. I haven't checked the source difference to 2.3.x I must
> admit.
>
>> I immagine that one of the reasons dovecot sends softCommits is because
>> without autoindex active and even if mailboxes are periodically indexed
>> from cron, the last emails received with be indexed at the moment of the
>> search.? 
> I expect that dovecot has to because of it's default behavior by only
> bringing the index up-to-date just before search. So it has towait for
> the index result to be available if there's been any new mails indexed.
>
>> 1) a configurable batch size would enable to tune the number of emails
>> per request and help stay under the 60 seconds hard coded http request
>> timeout. A configurable http timeout would be less useful, since this
>> will potentially run into other timeouts on solr side.
> Being able to configure it is great.
> But I don't think it solves much. I recompiled with 100 as batch size
> and it still ended in timeouts.
> Then I recompiled with 10min timeout and now I see all the batches
> completing and their processesing time is mostly between 1 and 2 minutes
> (so all would have failed).
>
> To me it looks like Solr just takes too long time to index. This is no
> small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
> and for this test it's not doing anything else, so I'm a bit surprised
> that even with only a few users this takes so long time.
>
> /Peter
>
>
Peter

I suppose you could go with a batch size of 50. If it's linear, you
could still keep under the default 60 seconds http request time :-)

I'm now testing with solr settings autoCommit 15 seconds, autoSoftCommit
60 seconds and sending no softCommits from dovecot and 500 batch size.

I've set up

    /usr/local/bin/doveadm index -A "*"

in crontab every 5 minutes so indexes will stay mostly up to date to
minimize amount of mail not already visible in the index when searches
are done.

The solr server is a small test virtual machine with 0.2 (shared) vCPU
and 0.6MB of memory and non SSD storage. It can index around 2000 emails
per minute when there is no other activity. Average email size is about
45Kb. I'm not indexing attachments.

John

Re: [PATCH] Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 16:04, Aki Tuomi via dovecot wrote:
>
>> On 14 April 2019 16:59 John Fawcett via dovecot < dovecot@dovecot.org
>> <mailto:dovecot@dovecot.org>> wrote:
>>
>>
>> On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
>>> On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:
>>>> If this value was made configurable people could set it to what they
>>>> want. However the underlying problem is likely on solr configuration.
>>> The Jetty that is included in Solr has its idle timeout set to 50
>>> seconds.  But in practice, I have not seen this timeout trigger ...
>>> and if the OP is seeing a 60 second timeout, then the 50 second idle
>>> timeout in Jetty must not be occurring.
>>> There may be a socket timeout configured on inter-server requests --
>>> distributed queries or the load balancing that SolrCloud does.  I can
>>> never remember whether this is the case by default.  I think it is.
>> >> If there is an issue on initial indexing, where you are not really
>> >> concerned about qucik visibility but just getting things into the
>> index
>> >> efficiently, a better approach would be for dovecot plugin not to
>> send
>> >> any commit or softCommit (or waitSearcher either) and that should
>> speed
>> >> things up. You'd need to configure solr with a long autoSoftCommit
>> >> maxTime and a reasonable autoCommit maxTime, which you could then
>> >> reconfigure when the load was done.
>> >
>>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>>> false on the autoCommit.  The autoSoftCommit setting is not enabled by
>>> default, but depending on how the index was created, Solr might try to
>>> set autoSoftCommit to 3 seconds ... which is WAY too short.
>>> I will usually increase the autoCommit time to 60 seconds, just to
>>> reduce the amount of work that Solr is doing.  The autoSoftCommit
>>> time, if it is used, should be set to a reasonably long value ...
>>> values between two and five minutes would be good.  Attempting to use
>>> a very short autoSoftCommit time will usually lead to problems.
>>> This thread says that dovecot is sending explicit commits.  One thing
>>> that might be happening to exceed 60 seconds is an extremely long
>>> commit, which is usually caused by excessive cache autowarming, but
>>> might be related to insufficient memory.  The max heap setting on an
>>> out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY
>>> small, and it doesn't take much index data before a much larger heap
>>> is required.
>>> Thanks,
>>> Shawn
>> I looked into the code (version 2.3.5.1): the fts-solr plugin is not
>> sending softCommit every 1000 emails. Emails from a single folder are
>> batched in up to a maximum of 1000 emails per request, but the
>> softCommit gets sent once per mailbox folder at the end of all the
>> requests for that folder.
>>
>> I immagine that one of the reasons dovecot sends softCommits is because
>> without autoindex active and even if mailboxes are periodically indexed
>> from cron, the last emails received with be indexed at the moment of the
>> search.  So while sending softCommit has the advantage of including
>> recent mails in searches, it means that softCommits are being done upon
>> user request. Frequency depends on user activity.
>>
>> Going back to the original problem: seems the first advice to Peter is
>> to look into solr configuration as others have said.
>>
>> From dovecot point of view I can see the following as potentially useful
>> features:
>>
>> 1) a configurable batch size would enable to tune the number of emails
>> per request and help stay under the 60 seconds hard coded http request
>> timeout. A configurable http timeout would be less useful, since this
>> will potentially run into other timeouts on solr side.
>>
>> 2) abilty to turn off softCommits so as to have a more predictable
>> softCommit workload. In that case autoSoftCommit should be configured in
>> solr. In order to minimize risk of recent emails not appearing in search
>> results, periodic indexing could be set up by cron.
>>
>> I've attached a patch, any comments are welcome (especially about
>> getting settings from the backend context).
>>
>> Example config
>>
>> plugin {
>>   fts = solr
>>   fts_solr =
>> url= https://user:passw...@solr.example.com:443/solr/dovecot/
>> batch_size=500 no_soft_commit
>> }
>>
>> John
>
> Can you please open a pull request to https://github.com/dovecot/core ?
> ---
> Aki Tuomi

Done, thanks for considering it.

John

[PATCH] Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
> On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:
>> If this value was made configurable people could set it to what they
>> want. However the underlying problem is likely on solr configuration.
>
> The Jetty that is included in Solr has its idle timeout set to 50
> seconds.  But in practice, I have not seen this timeout trigger ...
> and if the OP is seeing a 60 second timeout, then the 50 second idle
> timeout in Jetty must not be occurring.
>
> There may be a socket timeout configured on inter-server requests --
> distributed queries or the load balancing that SolrCloud does.  I can
> never remember whether this is the case by default.  I think it is.
>
>> If there is an issue on initial indexing, where you are not really
>> concerned about qucik visibility but just getting things into the index
>> efficiently, a better approach would be for dovecot plugin not to send
>> any commit or softCommit (or waitSearcher either) and that should speed
>> things up. You'd need to configure solr with a long autoSoftCommit
>> maxTime and a reasonable autoCommit maxTime, which you could then
>> reconfigure when the load was done.
>
> Solr ships with autoCommit set to 15 seconds and openSearcher set to
> false on the autoCommit.  The autoSoftCommit setting is not enabled by
> default, but depending on how the index was created, Solr might try to
> set autoSoftCommit to 3 seconds ... which is WAY too short.
>
> I will usually increase the autoCommit time to 60 seconds, just to
> reduce the amount of work that Solr is doing.  The autoSoftCommit
> time, if it is used, should be set to a reasonably long value ...
> values between two and five minutes would be good.  Attempting to use
> a very short autoSoftCommit time will usually lead to problems.
>
> This thread says that dovecot is sending explicit commits.  One thing
> that might be happening to exceed 60 seconds is an extremely long
> commit, which is usually caused by excessive cache autowarming, but
> might be related to insufficient memory.  The max heap setting on an
> out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY
> small, and it doesn't take much index data before a much larger heap
> is required.
>
> Thanks,
> Shawn

I looked into the code (version 2.3.5.1): the fts-solr plugin is not
sending softCommit every 1000 emails. Emails from a single folder are
batched in up to a maximum of 1000 emails per request, but the
softCommit gets sent once per mailbox folder at the end of all the
requests for that folder.

I immagine that one of the reasons dovecot sends softCommits is because
without autoindex active and even if mailboxes are periodically indexed
from cron, the last emails received with be indexed at the moment of the
search.  So while sending softCommit has the advantage of including
recent mails in searches, it means that softCommits are being done upon
user request. Frequency depends on user activity.

Going back to the original problem: seems the first advice to Peter is
to look into solr configuration as others have said.

>From dovecot point of view I can see the following as potentially useful
features:

1) a configurable batch size would enable to tune the number of emails
per request and help stay under the 60 seconds hard coded http request
timeout. A configurable http timeout would be less useful, since this
will potentially run into other timeouts on solr side.

2) abilty to turn off softCommits so as to have a more predictable
softCommit workload. In that case autoSoftCommit should be configured in
solr. In order to minimize risk of recent emails not appearing in search
results, periodic indexing could be set up by cron.

I've attached a patch, any comments are welcome (especially about
getting settings from the backend context).

Example config

plugin {
  fts = solr
  fts_solr =
url=https://user:passw...@solr.example.com:443/solr/dovecot/
batch_size=500 no_soft_commit
}

John

--- src/plugins/fts-solr/fts-solr-plugin.h.orig 2019-04-14 15:12:07.694289402 
+0200
+++ src/plugins/fts-solr/fts-solr-plugin.h  2019-04-14 14:04:17.213939414 
+0200
@@ -12,8 +12,10 @@
 
 struct fts_solr_settings {
const char *url, *default_ns_prefix;
+   unsigned int batch_size;
bool use_libfts;
bool debug;
+   bool no_soft_commit;
 };
 
 struct fts_solr_user {
--- src/plugins/fts-solr/fts-solr-plugin.c.orig 2019-04-14 11:41:03.591782439 
+0200
+++ src/plugins/fts-solr/fts-solr-plugin.c  2019-04-14 14:37:46.059433864 
+0200
@@ -10,6 +10,8 @@
 #include "fts-solr-plugin.h"
 
 
+#define DEFAULT_SOLR_BATCH_SIZE 1000
+
 const char *fts_solr_plugin_version = DOVECOT_ABI_VERSION;
 struct http_client *solr_http_client = NULL;
 
@@ -37,6 +39,10 @@
} else if (str_begins(*tmp, &qu

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 12/04/2019 12:09, Peter Mogensen via dovecot wrote:
> Looking further at tcpdumps of the Dovecot->Solr traffic and Solr
> metrics it doesn't seem like there's anything suspicious apart from the
> TCP windows running full and Dovecot backing of ... until it times out
> and close the connection.
>
> From my understanding of how Dovecot operates towards Solr it will flush
> ~1000 documents towards Solr in /update request until it has traversed
> the mailbox (let's say 20.000 mails), doing softCommits after each.
>
> But is it really reasonable for Dovecot to expect that no request will
> take more than 60s to process by Solr?
> It doesn't seem like my Solr can handle that, although it does process
> documents and it does reasonably fast clear pending documents after
> Dovecot closes the connection.
>
> On the surface it looks like Dovecot is too impatient.
>
> /Peter

If this value was made configurable people could set it to what they
want. However the underlying problem is likely on solr configuration.

Is this a problem only on initial indexing or an ongoing problem after
initial indexing?

The parameters that the solr plugin are using are designed to make
documents visible to searches quickly.

If there is an issue on initial indexing, where you are not really
concerned about qucik visibility but just getting things into the index
efficiently, a better approach would be for dovecot plugin not to send
any commit or softCommit (or waitSearcher either) and that should speed
things up. You'd need to configure solr with a long autoSoftCommit
maxTime and a reasonable autoCommit maxTime, which you could then
reconfigure when the load was done.

If you're using dovecot built from source code it should be possible to
test that by some minor modification of the code in  fts-backend-solr.c.

John

Re: Secure Client-Initiated Renegotiation

[John Fawcett via dovecot]

On 11/04/2019 23:28, sergio via dovecot wrote:
> Hello.
>
> I've just tested my system that runs dovecot 2.3.4.1 on debian buster
> with testssl.sh (https://testssl.sh/) and is says:
>
> Secure Renegotiation (CVE-2009-3555)    not vulnerable (OK)
> Secure Client-Initiated Renegotiation   VULNERABLE (NOT ok), potential
> DoS threat
>
> Is this a configuration or a compilation issue and how to solve it?
>
This should be interpreted as meaning that client initiated
renegotiation is enabled. The tool does not test whether the software
mitigates the dos threat by limiting the number of negotiations to a
configurable limit.

Having said that, I think that mitigating this in dovecot would require
a code change. What I'm not sure about is whether the best route would
be to turn off client side renegotiation or only limit it. A previous
version of openssl turned it off but then it was re-introduced. That
would require further investigation to understand the best solution.

John

Re: failed: read(/var/run/dovecot/dns-client)

[John Fawcett via dovecot]

On 11/04/2019 22:09, Laura Smith via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Thursday, April 11, 2019 9:01 PM, John Fawcett via dovecot 
>  wrote:
>
>> On 11/04/2019 10:02, Laura Smith via dovecot wrote:
>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Thursday, April 11, 2019 12:55 AM, John Fawcett via dovecot 
>>> dovecot@dovecot.org wrote:
>>>
>>>> On 11/04/2019 00:51, Laura Smith via dovecot wrote:
>>>>
>>>>> ‐‐‐ Original Message ‐‐‐
>>>>> On Wednesday, April 10, 2019 11:48 PM, John Fawcett via dovecot 
>>>>> dovecot@dovecot.org wrote:
>>>>>
>>>>>> On 11/04/2019 00:18, Laura Smith via dovecot wrote:
>>>>>>
>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>> On Wednesday, April 10, 2019 10:24 PM, Aki Tuomi 
>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>
>>>>>>>>> On 10 April 2019 23:56 Laura Smith via dovecot < dovecot@dovecot.org> 
>>>>>>>>> wrote:
>>>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>>>> On Wednesday, April 10, 2019 9:14 PM, Aki Tuomi < 
>>>>>>>>> aki.tu...@open-xchange.com> wrote:
>>>>>>>>>
>>>>>>>>>>> On 10 April 2019 23:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>>>> wrote:
>>>>>>>>>>> Sent with ProtonMail Secure Email.
>>>>>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>>>>>> On Wednesday, April 10, 2019 8:20 PM, Aki Tuomi 
>>>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> On 10 April 2019 22:13 Laura Smith via dovecot 
>>>>>>>>>>>>> dovecot@dovecot.org wrote:
>>>>>>>>>>>>> On Wednesday, April 10, 2019 7:57 PM, Aki Tuomi 
>>>>>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 10 April 2019 21:26 Laura Smith via dovecot 
>>>>>>>>>>>>>>> dovecot@dovecot.org wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ==
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> dsync( foo...@example.com): Error: 
>>>>>>>>>>>>>>> imapc(foobar.example.com:993): dns_lookup(foobar.example.com) 
>>>>>>>>>>>>>>> failed: read(/var/run/dovecot/dns-client) failed: 
>>>>>>>>>>>>>>> read(size=512) failed: Connection reset by peer
>>>>>>>>>>>>>>> This is dovecot's internal dns-client, and something goes wrong 
>>>>>>>>>>>>>>> when talking to the service.
>>>>>>>>>>>>>>> dsync( foo...@example.com): Error: Failed to initialize user: 
>>>>>>>>>>>>>>> imapc: Login to foobar.example.com failed: Disconnected from 
>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>> This is btw dsync service, not imap service.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ===
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Initially I thought "oh no, not another AppArmor block".
>>>>>>>>>>>>>>> But then surely the second message would not appear if the DNS 
>>>>>>>>>>>>>>> lookup was not successful ?
>>>>>>

Re: failed: read(/var/run/dovecot/dns-client)

[John Fawcett via dovecot]

On 11/04/2019 10:02, Laura Smith via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Thursday, April 11, 2019 12:55 AM, John Fawcett via dovecot 
>  wrote:
>
>> On 11/04/2019 00:51, Laura Smith via dovecot wrote:
>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Wednesday, April 10, 2019 11:48 PM, John Fawcett via dovecot 
>>> dovecot@dovecot.org wrote:
>>>
>>>> On 11/04/2019 00:18, Laura Smith via dovecot wrote:
>>>>
>>>>> ‐‐‐ Original Message ‐‐‐
>>>>> On Wednesday, April 10, 2019 10:24 PM, Aki Tuomi 
>>>>> aki.tu...@open-xchange.com wrote:
>>>>>
>>>>>>> On 10 April 2019 23:56 Laura Smith via dovecot < dovecot@dovecot.org> 
>>>>>>> wrote:
>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>> On Wednesday, April 10, 2019 9:14 PM, Aki Tuomi < 
>>>>>>> aki.tu...@open-xchange.com> wrote:
>>>>>>>
>>>>>>>>> On 10 April 2019 23:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>> wrote:
>>>>>>>>> Sent with ProtonMail Secure Email.
>>>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>>>> On Wednesday, April 10, 2019 8:20 PM, Aki Tuomi 
>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>
>>>>>>>>>>> On 10 April 2019 22:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>>>> wrote:
>>>>>>>>>>> On Wednesday, April 10, 2019 7:57 PM, Aki Tuomi 
>>>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> On 10 April 2019 21:26 Laura Smith via dovecot 
>>>>>>>>>>>>> dovecot@dovecot.org wrote:
>>>>>>>>>>>>> ==
>>>>>>>>>>>>> dsync( foo...@example.com): Error: imapc(foobar.example.com:993): 
>>>>>>>>>>>>> dns_lookup(foobar.example.com) failed: 
>>>>>>>>>>>>> read(/var/run/dovecot/dns-client) failed: read(size=512) failed: 
>>>>>>>>>>>>> Connection reset by peer
>>>>>>>>>>>>> This is dovecot's internal dns-client, and something goes wrong 
>>>>>>>>>>>>> when talking to the service.
>>>>>>>>>>>>> dsync( foo...@example.com): Error: Failed to initialize user: 
>>>>>>>>>>>>> imapc: Login to foobar.example.com failed: Disconnected from 
>>>>>>>>>>>>> server
>>>>>>>>>>>>> This is btw dsync service, not imap service.
>>>>>>>>>>>>> ===
>>>>>>>>>>>>> Initially I thought "oh no, not another AppArmor block".
>>>>>>>>>>>>> But then surely the second message would not appear if the DNS 
>>>>>>>>>>>>> lookup was not successful ?
>>>>>>>>>>>>> Also "dig foobar.example.com" works fine.
>>>>>>>>>>>>> How should I be troubleshooting this ? And if it is still likely 
>>>>>>>>>>>>> to be AppArmor, what is calling it ? "doveadm" itself or 
>>>>>>>>>>>>> something else ? What does "/var/run/dovecot/dns-client" do and 
>>>>>>>>>>>>> why doesn't dovecot use standard OS calls like everyone else ?
>>>>>>>>>>>>> Because the "standard OS call" is blocking and we would prefer it 
>>>>>>>>>>>>> to not block everything else.
>>>>>>>>>>>>> So many questions !
>>>>

Re: failed: read(/var/run/dovecot/dns-client)

[John Fawcett via dovecot]

On 11/04/2019 00:51, Laura Smith via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Wednesday, April 10, 2019 11:48 PM, John Fawcett via dovecot 
>  wrote:
>
>> On 11/04/2019 00:18, Laura Smith via dovecot wrote:
>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Wednesday, April 10, 2019 10:24 PM, Aki Tuomi aki.tu...@open-xchange.com 
>>> wrote:
>>>
>>>>> On 10 April 2019 23:56 Laura Smith via dovecot < dovecot@dovecot.org> 
>>>>> wrote:
>>>>> ‐‐‐ Original Message ‐‐‐
>>>>> On Wednesday, April 10, 2019 9:14 PM, Aki Tuomi < 
>>>>> aki.tu...@open-xchange.com> wrote:
>>>>>
>>>>>>> On 10 April 2019 23:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>> wrote:
>>>>>>> Sent with ProtonMail Secure Email.
>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>> On Wednesday, April 10, 2019 8:20 PM, Aki Tuomi 
>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>
>>>>>>>>> On 10 April 2019 22:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>> wrote:
>>>>>>>>> On Wednesday, April 10, 2019 7:57 PM, Aki Tuomi 
>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>
>>>>>>>>>>> On 10 April 2019 21:26 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> ==
>>>>>>>>>>>
>>>>>>>>>>> dsync( foo...@example.com): Error: imapc(foobar.example.com:993): 
>>>>>>>>>>> dns_lookup(foobar.example.com) failed: 
>>>>>>>>>>> read(/var/run/dovecot/dns-client) failed: read(size=512) failed: 
>>>>>>>>>>> Connection reset by peer
>>>>>>>>>>> This is dovecot's internal dns-client, and something goes wrong 
>>>>>>>>>>> when talking to the service.
>>>>>>>>>>> dsync( foo...@example.com): Error: Failed to initialize user: 
>>>>>>>>>>> imapc: Login to foobar.example.com failed: Disconnected from server
>>>>>>>>>>> This is btw dsync service, not imap service.
>>>>>>>>>>>
>>>>>>>>>>> ===
>>>>>>>>>>>
>>>>>>>>>>> Initially I thought "oh no, not another AppArmor block".
>>>>>>>>>>> But then surely the second message would not appear if the DNS 
>>>>>>>>>>> lookup was not successful ?
>>>>>>>>>>> Also "dig foobar.example.com" works fine.
>>>>>>>>>>> How should I be troubleshooting this ? And if it is still likely to 
>>>>>>>>>>> be AppArmor, what is calling it ? "doveadm" itself or something 
>>>>>>>>>>> else ? What does "/var/run/dovecot/dns-client" do and why doesn't 
>>>>>>>>>>> dovecot use standard OS calls like everyone else ?
>>>>>>>>>>> Because the "standard OS call" is blocking and we would prefer it 
>>>>>>>>>>> to not block everything else.
>>>>>>>>>>> So many questions !
>>>>>>>>>>> Aki
>>>>>>>>>>> Thanks for your reply, but both those message are generated from a 
>>>>>>>>>>> simple :
>>>>>>>>>>> doveadm -v -o mail_fsync=never backup -R -u foo...@example.com 
>>>>>>>>>>> imapc:
>>>>>>>>>>> So I don't know what you mean about dsync service failing ? Surely 
>>>>>>>>

Re: failed: read(/var/run/dovecot/dns-client)

[John Fawcett via dovecot]

On 11/04/2019 00:18, Laura Smith via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Wednesday, April 10, 2019 10:24 PM, Aki Tuomi  
> wrote:
>
>>> On 10 April 2019 23:56 Laura Smith via dovecot < dovecot@dovecot.org> wrote:
>>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Wednesday, April 10, 2019 9:14 PM, Aki Tuomi < 
>>> aki.tu...@open-xchange.com> wrote:
>>>
> On 10 April 2019 23:13 Laura Smith via dovecot dovecot@dovecot.org wrote:
> Sent with ProtonMail Secure Email.
> ‐‐‐ Original Message ‐‐‐
> On Wednesday, April 10, 2019 8:20 PM, Aki Tuomi 
> aki.tu...@open-xchange.com wrote:
>>> On 10 April 2019 22:13 Laura Smith via dovecot dovecot@dovecot.org 
>>> wrote:
>>> On Wednesday, April 10, 2019 7:57 PM, Aki Tuomi 
>>> aki.tu...@open-xchange.com wrote:
> On 10 April 2019 21:26 Laura Smith via dovecot dovecot@dovecot.org 
> wrote:
> ==
> dsync( foo...@example.com): Error: imapc(foobar.example.com:993): 
> dns_lookup(foobar.example.com) failed: 
> read(/var/run/dovecot/dns-client) failed: read(size=512) failed: 
> Connection reset by peer
 This is dovecot's internal dns-client, and something goes wrong when 
 talking to the service.
> dsync( foo...@example.com): Error: Failed to initialize user: imapc: 
> Login to foobar.example.com failed: Disconnected from server
 This is btw dsync service, not imap service.
> ===
> Initially I thought "oh no, not another AppArmor block".
> But then surely the second message would not appear if the DNS lookup 
> was not successful ?
> Also "dig foobar.example.com" works fine.
> How should I be troubleshooting this ? And if it is still likely to 
> be AppArmor, what is calling it ? "doveadm" itself or something else 
> ? What does "/var/run/dovecot/dns-client" do and why doesn't dovecot 
> use standard OS calls like everyone else ?
 Because the "standard OS call" is blocking and we would prefer it to 
 not block everything else.
> So many questions !
 Aki
>>> Thanks for your reply, but both those message are generated from a 
>>> simple :
>>> doveadm -v -o mail_fsync=never backup -R -u foo...@example.com imapc:
>>> So I don't know what you mean about dsync service failing ? Surely the 
>>> DNS lookup succeeded if the 'dsync service' failed due to remote 
>>> disconnect ?
>>> I'm still none the wiser as to where to start looking for 
>>> troubleshoting ?
>> Did you check dovecot logs? Maybe there is something useful?
>> Aki
> Only the same old cryptic message about dns-client ?
> master: Fatal: execv(/usr/lib/dovecot/dns-client) failed: Permission 
> denied
 Something prevents executing the dns-client binary.
> master: Error: service(dns_client): command startup failed, throttling 
> for 16 secs
> dns_client: Fatal: master: service(dns_client): child 14293 returned 
> error 84 (exec() failed)
 Aki
>>> Yes but is it being called by doveadm directly or by some other dovecot 
>>> program ? If I'm going to have to go down the AppArmor route, then I would 
>>> prefer if you told me what was calling it instead of me having to 
>>> un-necessarily spend time doing straces !
>>>
>>> Also, should I be able to call dns-client directly myself ? (or is there a 
>>> way to do so to enable testing ?
>> It is started by dovecot's master process when you connect to dns-client 
>> unix socket. You can try
>>
>> socat stdio unix-connect:/var/run/dovecot/dns-client
>>
>> I thought apparmor tells when something is blocked into kernel log? have you 
>> checked dmesg?
>>
>> Apologies for your frustration. 
>> ---
> Yeah nothing in dmesg.  I'm still hunting around to find some log somewhere 
> but so far silence.
>
> "socat stdio unix-connect:/var/run/dovecot/dns-client" runs but returns 
> nothing. Is that expected ?
>
> When you say "dovecot's master process", so  doveadm sync talks to the master 
> process ?  So in terms of apparmor I would therefore be looking at 
> /usr/sbin/dovecot ?  If that's the case, the relevant apparmor permisssions 
> are already provided :
>   /{,var/}run/dovecot/ rw,
>   /{,var/}run/dovecot/** rw,

Laura

Do the above apparmor settings give permission to dovecot to execute
/usr/lib/dovecot/dns-client, assuming that the user under which dovecot
is running already has file system permissions to do that?

John

Re: Where to report (potential) Dovecot bugs

[John Fawcett via dovecot]

On 09/04/2019 22:03, Mark Moseley via dovecot wrote:
> I'm curious if this is still the right place to report potential bugs
> with Dovecot. 
>
> Is there a Dovecot bug tracker somewhere?

https://www.dovecot.org/bugreport.html

Re: dovecot[30008]: auth-worker: Fatal: master: service(auth-worker): child 31074 killed with signal 11

[John Fawcett via dovecot]

On 08/04/2019 16:22, Herbert via dovecot wrote:
> Hi
>
> Since a recent DNF update, I see many log file entries like the following:
>
> Apr 08 16:16:36 udoo.symmetrix.ch audit[31074]: ANOM_ABEND
> auid=4294967295 uid=0 gid=0 ses=4294967295
> subj=system_u:system_r:dovecot_auth_t:s0 pid=31074 comm="auth"
> exe="/usr/libexec/dovecot/auth" sig=11 res=1
> Apr 08 16:16:36 udoo.symmetrix.ch kernel: traps: auth[31074] general
> protection fault ip:7ff45b5f98a7 sp:7fff0df96830 error:0 in
> libmariadb.so.3[7ff45b5d7000+46000]
> Apr 08 16:16:36 udoo.symmetrix.ch dovecot[30008]: auth-worker: Fatal:
> master: service(auth-worker): child 31074 killed with signal 11 (core
> not dumped - https://dovecot.org/bugreport.html#coredumps)
> Apr 08 16:16:38 udoo.symmetrix.ch abrt-dump-journal-oops[1139]:
> abrt-dump-journal-oops: Found oopses: 1
> Apr 08 16:16:38 udoo.symmetrix.ch abrt-dump-journal-oops[1139]:
> abrt-dump-journal-oops: Creating problem directories
> Apr 08 16:16:39 udoo.symmetrix.ch abrt-dump-journal-oops[1139]:
> Reported 1 kernel oopses to Abrt
> Apr 08 16:16:39 udoo.symmetrix.ch abrt-server[31104]: Can't find a
> meaningful backtrace for hashing in '.'
> Apr 08 16:16:39 udoo.symmetrix.ch abrt-server[31104]: Option
> 'DropNotReportableOopses' is not configured
> Apr 08 16:16:39 udoo.symmetrix.ch abrt-server[31104]: Preserving oops
> '.' because DropNotReportableOopses is 'no'
> Apr 08 16:16:39 udoo.symmetrix.ch dbus-daemon[1022]: [system]
> Activating service name='org.freedesktop.problems' requested by
> ':1.779' (uid=0 pid=31122 comm="/usr/bin/python3
> /usr/bin/abrt-action-notify -d /v"
> label="system_u:system_r:abrt_t:s0-s0:c0.c1023") (using servicehelper)
> Apr 08 16:16:40 udoo.symmetrix.ch dbus-daemon[1022]: [system]
> Successfully activated service 'org.freedesktop.problems'
> Apr 08 16:16:40 udoo.symmetrix.ch abrt-notification[31129]: System
> encountered a non-fatal error in ??()
>
>
> # uname -a
> Linux udoo.symmetrix.ch 5.0.5-100.fc28.x86_64 #1 SMP Wed Mar 27
> 22:16:29 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> # dovecot --version
> 2.3.5 (513208660)
>
> # mysql --version
> mysql  Ver 15.1 Distrib 10.2.22-MariaDB, for Linux (x86_64) using
> readline 5.1
>
> Unfortunately I didn't manage to get a core dump, sorry.
>
> Kind regards,
> Herbert
>
>
Herbert

it's just a guess without a backtrace, but it looks similar to an
already known issue of a double mysql_close. The code has not yet in a
release but it's in the master-2.3 branch. If you are building from
source you might be able to patch it yourself while waiting for the next
2.3.x release.

https://github.com/dovecot/core/commit/8d78721bbf86657966750da3bcfc7fe9a92c7417

John

Re: SMTPUTF8 support

[John Fawcett via dovecot]

On 06/04/2019 11:34, André Rodier via dovecot wrote:
> On Sat, 2019-04-06 at 11:19 +0200, John Fawcett via dovecot wrote:
>> On 06/04/2019 10:07, André Rodier via dovecot wrote:
>>> On 05/04/2019 08:57, David Bürgin via dovecot wrote:
>>>> André, are you quite sure you have it working?
>>>>
>>>> In this thread someone from Open-Xchange stated that no, Dovecot doesn’t
>>>> have SMTPUTF8 support implemented, and the same response was given by
>>>> another Dovecot developer last September (it ‘is being considered’ was
>>>> the answer then, see
>>>> https://dovecot.org/pipermail/dovecot/2018-September/112887.html).
>>>>
>>>> I am using LMTP to deliver mail to Dovecot from Postfix, and delivery
>>>> fails with Postfix reporting: ‘SMTPUTF8 is required, but was not offered
>>>> by host mail.my.org[private/dovecot-lmtp]’. I doubt that this can work
>>>> without a change in Dovecot LMTP.
>>>>
>>>> Anyway, I am also interested in SMTPUTF8 support, so +1 from me!
>>>>
>>>>
>>> Hello David et al,
>>>
>>> I have it working with my set up, and I confirm it is not easy.
>>>
>>> If I limit the stack to OpenLDAP, Postfix and Dovecot to their
>>> simplest configuration, it is relatively easy, even with slightly
>>> outdated software in Debian Stretch. 
>> I think you are confusing acceptance of UTF8 in the localpart of the
>> email address (which if I remember correctly worked in the 2.2 branch)
>> with SMTPUTF8 support. As mentioned in the thread above Dovecot does not
>> as yet support SMTPUTF8. Support for SMTPUTF8 would mean among other
>> things changing the source code of Dovecot so that it announces the
>> SMTPUTF8 key word to the upstream MTA.
>>
>> John
> Hello John,
>
> You are probably right.
>
> I am not well aware of the low level details and the terminology.
>
> I confess I am mostly interested in the proper reception of the emails
> with accentuated user names, which is already sometimes difficult to
> achieve.
>
> Is there any link where I can learn the communication rules between
> Dovecot and Postfix and the SMTPUTF8 standard ?
>
> Kind regards,
> André
>
André

Daniel Lange has posted some links to this topic from Postfix and
earlier discussions on the mailing list.

Postfix has this SMTPUTF8 documentation:

http://www.postfix.org/SMTPUTF8_README.html

On that page you can also find links to the relevant standards including
SMTPUTF8 standard.

John

Re: SMTPUTF8 support

[John Fawcett via dovecot]

On 06/04/2019 10:07, André Rodier via dovecot wrote:
> On 05/04/2019 08:57, David Bürgin via dovecot wrote:
>> André, are you quite sure you have it working?
>>
>> In this thread someone from Open-Xchange stated that no, Dovecot doesn’t
>> have SMTPUTF8 support implemented, and the same response was given by
>> another Dovecot developer last September (it ‘is being considered’ was
>> the answer then, see
>> https://dovecot.org/pipermail/dovecot/2018-September/112887.html).
>>
>> I am using LMTP to deliver mail to Dovecot from Postfix, and delivery
>> fails with Postfix reporting: ‘SMTPUTF8 is required, but was not offered
>> by host mail.my.org[private/dovecot-lmtp]’. I doubt that this can work
>> without a change in Dovecot LMTP.
>>
>> Anyway, I am also interested in SMTPUTF8 support, so +1 from me!
>>
>>
>
> Hello David et al,
>
> I have it working with my set up, and I confirm it is not easy.
>
> If I limit the stack to OpenLDAP, Postfix and Dovecot to their
> simplest configuration, it is relatively easy, even with slightly
> outdated software in Debian Stretch. 

I think you are confusing acceptance of UTF8 in the localpart of the
email address (which if I remember correctly worked in the 2.2 branch)
with SMTPUTF8 support. As mentioned in the thread above Dovecot does not
as yet support SMTPUTF8. Support for SMTPUTF8 would mean among other
things changing the source code of Dovecot so that it announces the
SMTPUTF8 key word to the upstream MTA.

John

Re: Upgrading to 2.3

[John Fawcett via dovecot]

On 09/03/2019 14:10, Christian Schmidt via dovecot wrote:
> Hi,
>
> @lbutlr via dovecot, 08.03.19:
>> On 8 Mar 2019, at 05:54, Aki Tuomi via dovecot 
>> wrote:
>>> https://wiki.dovecot.org/Upgrading
>>
>> Duh. I wasn't looking for a URL that was specific.
>
> https://wiki2.dovecot.org/Upgrading/2.3 ;-)
>
> Kind Regards
> Christian
>
Noticed a small typo in the wiki for upgrading to 2.3

recipient_delimiters

should read

recipient_delimiter

John

Re: smtp_address_cmp does not ignore case of localpart

[John Fawcett via dovecot]

On 12/02/2019 21:41, Andreas Jobs via dovecot wrote:
> On Tue, Feb 12, 2019 at 08:12:43AM +0100, John Fawcett via dovecot wrote:
>> On 11/02/2019 22:43, Andreas Jobs via dovecot wrote:
>> In practice if you convert the username part to lower case in the MTA
>> then you will accept mail with mixed case but only have to deal with
>> lowercase in Dovecot.
>
> Yes, but the vacation action (also) compares :addresses against the
> header to|cc|sender... And changing the headers is a bad idea.
>
> Maybe my subject line was a bad choice :-)
>
>
> --Andreas
>
You're right, ignore my suggestion. Only the domain is treated case
insensitive and localpart is compared case sensitive.

The change in itself looks trivial, but it would affect more than
vacation (smtp_address_equals is used in a few places in dovecot and
pigeonhole)

John

Re: smtp_address_cmp does not ignore case of localpart

[John Fawcett via dovecot]

On 11/02/2019 22:43, Andreas Jobs via dovecot wrote:
> Hi,
>
> in src/lib-smtp/smtp-address.c line 759 the localpart is compared with
> null_strcmp, which does not ignore the case. While this is the correct
> behavior (according to the RFC) it causes problems in real life.
>
> In my case the vacation notices are not sent, if someone writes to
> first.l...@ruhr-uni-bochum.de because :addresses contains (only)
> first.l...@ruhr-uni-bochum.de.
>
>
>
> Regards,
> Andreas
>
In practice if you convert the username part to lower case in the MTA
then you will accept mail with mixed case but only have to deal with
lowercase in Dovecot.

John