Re: Failing to archive many messages

[Francis Augusto Medeiros-Logeay via dovecot]

Thanks a lot Marc.

I actually found something that works for me:

I replaced `sa-learn -u xx --ham`
with
`spamc -u xx -L ham`

Archiving wasn't as fast as simply moving messages, but was considerably 
faster - fast enough that no errors on Roundcube are reported.


Best,
Francis

---
Francis Augusto Medeiros-Logeay
Oslo, Norway

On 2024-05-18 12:06, Marc wrote:
Today I realized that it was indeed the sa-learn that is slowing down 
my

Archiving.

I added an «&» on the end of the line that calls the sa-learn (called 
by

the report-ham.sieve script):

exec /usr/bin/sa-learn -u ${1} --ham  &

Does anyone thing this is a bad idea? I noticed that archiving got way
faster this way…


You fork/spawn a new process. So depending on how many messages are 
moved, that many processes are created. I am not using sa-learn, but I 
think this could be a memory consuming task. So manybe check for OOM 
crashes or your swap space slowing everything down.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Failing to archive many messages

[Francis Augusto Medeiros-Logeay via dovecot]

Today I realized that it was indeed the sa-learn that is slowing down my 
Archiving.

I added an «&» on the end of the line that calls the sa-learn (called by the 
report-ham.sieve script):

exec /usr/bin/sa-learn -u ${1} --ham  &

Does anyone thing this is a bad idea? I noticed that archiving got way faster 
this way…

Best,

Francis


> On 9 May 2024, at 23:09, dovecot-requ...@dovecot.org wrote:
> 
>> On 9 May 2024, at 19:45, dovecot-requ...@dovecot.org 
>> <mailto:dovecot-requ...@dovecot.org> wrote:
>> 
>> Francis Augusto Medeiros-Logeay via dovecot skrev den 2024-05-09 12:29:
>> 
>>>> If you want to analyze emails, why not do it with scheduled crons 
>>>> after they are archived?
>> 
>> maybe ask for skip older then one day in here 
>> https://github.com/isbg/isbg/tree/master
>> 
>> while roundcube just set the needed flag for retest ?
>> 
>>> wouldn't it reanalyze all archived messages anyway?
>> 
>> sometimes email is ham at recpt stage, while its spam later on test, 
>> this is why isbg is made imho :)
> 
> 
> I see.
> 
> But I wonder if there is something wrong with what I already have. It worked 
> fine for years, and it works fine for most of my users. I tested with a few 
> accounts, and Archving (with its sa-learn script) works blazingly fast. But 
> with one user - happens to be mine - it is super slow when archiving, most 
> likely due the sa-learn.
> 
> Best,
> 
> Francis

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Failing to archive many messages

[Francis Augusto Medeiros-Logeay via dovecot]

> On 9 May 2024, at 19:45, dovecot-requ...@dovecot.org wrote:
> 
> Francis Augusto Medeiros-Logeay via dovecot skrev den 2024-05-09 12:29:
> 
>>> If you want to analyze emails, why not do it with scheduled crons 
>>> after they are archived?
> 
> maybe ask for skip older then one day in here 
> https://github.com/isbg/isbg/tree/master
> 
> while roundcube just set the needed flag for retest ?
> 
>> wouldn't it reanalyze all archived messages anyway?
> 
> sometimes email is ham at recpt stage, while its spam later on test, 
> this is why isbg is made imho :)


I see.

But I wonder if there is something wrong with what I already have. It worked 
fine for years, and it works fine for most of my users. I tested with a few 
accounts, and Archving (with its sa-learn script) works blazingly fast. But 
with one user - happens to be mine - it is super slow when archiving, most 
likely due the sa-learn.

Best,

Francis
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Authentication questions

[Francis Augusto Medeiros-Logeay via dovecot]

> On 9 May 2024, at 19:45, Aki Tuomi  wrote:
> 
>  
>> On 09/05/2024 20:15 EEST Francis Augusto Medeiros-Logeay via dovecot 
>> mailto:dovecot@dovecot.org>> wrote:
>>  
>>  
>> Hi,
>>  
>> I was wondering:
>>  
>> 1 - Is it possible to configure authentication methods per user? For 
>> example, oauth2 for most users, but plain for others?
>> 2 - I had a feeling that when oauth2 authentication fails, dovecot tries to 
>> authenticate via plain with the received token. Doesn’t seem logical, but I 
>> get my user blocked on my directory server (freeipa) after a few failed oath 
>> authentications. If so, can this be prevented?
>>  
>> Best,
>>  
>> Francis
>> ___
>> dovecot mailing list -- dovecot@dovecot.org <mailto:dovecot@dovecot.org>
>> To unsubscribe send an email to dovecot-le...@dovecot.org 
>> <mailto:dovecot-le...@dovecot.org> 
> Yep. See 
> https://doc.dovecot.org/configuration_manual/authentication/password_databases_passdb/#passdb-setting
>  
> you can filter by mechanism.
>  
> Aki

The weird thing is that I get this still: 

May 09 21:45:47 auth: Error: oauth2(myu...@mydomain.com,48.237.124.127): oauth2 
failed: Introspection failed: No username returned
May 09 21:45:47 auth: Error: ldap(myu...@mydomain.com,48.237.124.127): 
ldap_bind() failed: Constraint violation

Even when I have my configuration like this:

auth_mechanisms = $auth_mechanisms xoauth2 oauthbearer


passdb {
  driver = oauth2
  mechanisms = xoauth2 oauthbearer
  args = /etc/dovecot/dovecot-oauth2.conf.ext
  result_failure=return-fail
}

What could be the cause? 

Best,
Francis 


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Authentication questions

[Francis Augusto Medeiros-Logeay via dovecot]

> On 9 May 2024, at 19:45, Aki Tuomi  wrote:
> 
>  
>> On 09/05/2024 20:15 EEST Francis Augusto Medeiros-Logeay via dovecot 
>> mailto:dovecot@dovecot.org>> wrote:
>>  
>>  
>> Hi,
>>  
>> I was wondering:
>>  
>> 1 - Is it possible to configure authentication methods per user? For 
>> example, oauth2 for most users, but plain for others?
>> 2 - I had a feeling that when oauth2 authentication fails, dovecot tries to 
>> authenticate via plain with the received token. Doesn’t seem logical, but I 
>> get my user blocked on my directory server (freeipa) after a few failed oath 
>> authentications. If so, can this be prevented?
>>  
>> Best,
>>  
>> Francis
>> ___
>> dovecot mailing list -- dovecot@dovecot.org <mailto:dovecot@dovecot.org>
>> To unsubscribe send an email to dovecot-le...@dovecot.org 
>> <mailto:dovecot-le...@dovecot.org> 
> Yep. See 
> https://doc.dovecot.org/configuration_manual/authentication/password_databases_passdb/#passdb-setting
>  
> you can filter by mechanism.
>  
> Aki

Thanks, this is great!

Best,
Francis

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Failing to archive many messages

[Francis Augusto Medeiros-Logeay via dovecot]

> On 9 May 2024, at 12:55, dovecot-requ...@dovecot.org wrote:
> 
> Francis Augusto Medeiros-Logeay via dovecot skrev den 2024-05-09 11:28:
> 
>> Actually, I don’t know here if that’s the sieve that’s slowing down the 
>> process, or if it is something else.
>> I see that yeah, sa-learn takes a little while to execute. But maybe 
>> the issue is that Dovecot doesn’t seem to
>> release the IMAP connection until each message went through sieve. This 
>> gives me some timeout on
>> Roundcube (it says just «server error»). All the messages are indeed 
>> archived.
>> 
>> Is that something here I could do better?
> 
> use tmpfs to store tmp content, or try solve it without pipe with imho 
> creates tmp files
> 
> https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/
> 
> try spamc

Thanks Benny. Using spamc and sa-learn isn’t a bit the same? But I’ll check the 
docs.

Bestm

Francis 

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Authentication questions

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

I was wondering:

1 - Is it possible to configure authentication methods per user? For example, 
oauth2 for most users, but plain for others?
2 - I had a feeling that when oauth2 authentication fails, dovecot tries to 
authenticate via plain with the received token. Doesn’t seem logical, but I get 
my user blocked on my directory server (freeipa) after a few failed oath 
authentications. If so, can this be prevented?

Best,

Francis 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Failing to archive many messages

[Francis Augusto Medeiros-Logeay via dovecot]

=

On 2024-05-09 12:03, Marc wrote:
I see report-ham, instead of archiving? I am not entirely sure if this 
is an efficient approach to archiving. My users can only set a flag for 
'auto archiving' via sieve and then 3 months after every year messages 
are being moved by cron jobs, and modification logs are put into their 
accounts. This all is only happening in 'dead hours'.


If you want to analyze emails, why not do it with scheduled crons after 
they are archived?




wouldn't it reanalyze all archived messages anyway?
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Failing to archive many messages

[Francis Augusto Medeiros-Logeay via dovecot]

> I noticed that my sieve rules are taking a very long time when archiving 
> messages. I can see on my logs that every message takes about two-three 
> seconds to be evaluated.
> 
> I also noticed this happens for a few accounts. Do you know what could be 
> causing this?
> 
> Best,
> Francis
> 

Actually, I don’t know here if that’s the sieve that’s slowing down the 
process, or if it is something else.
I see that yeah, sa-learn takes a little while to execute. But maybe the issue 
is that Dovecot doesn’t seem to release the IMAP connection until each message 
went through sieve. This gives me some timeout on Roundcube (it says just 
«server error»). All the messages are indeed archived.

Is that something here I could do better?

Best,
Francis 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: OAUTH2 problem when migrating to Ubuntu 24.04

[Francis Augusto Medeiros-Logeay via dovecot]

---
Francis Augusto Medeiros-Logeay
Oslo, Norway

On 2024-05-09 10:11, Aki Tuomi wrote:
On 09/05/2024 10:46 EEST Francis Augusto Medeiros-Logeay via dovecot 
 wrote:



Hi,

I have a Dovecot server on Ubuntu 22.04, which works fine with Oauth2. 
I am building a new container based on Ubuntu 24.04. IMAP seems to 
work fine with plain authentication, but oauth2 fails (Dovecot 
v2.3.21). Same configuration as before.


However, I got this message:

May 09 09:41:57 auth: Error: 
oauth2(f...@mydomain.eu,10.110.41.32,): oauth2 
failed: Introspection failed: No username returned


Weird, as it does indeed seems to get the username (as seen on the 
log).


Any clue on why this fails?

Best,

Francis


It means that introspection did not return username for the token. This 
is important, because if the token is not validated to belong to the 
user attempting to log in, anyone could login as anyone with any token.


Aki


Actually, the problem was not that. It was some change on Dovecot where 
suddenly I need to add "clientid:client_secret@" on the url (googled it, 
and saw that you advised that before). Now it works.




Best,
Francis
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

OAUTH2 problem when migrating to Ubuntu 24.04

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

I have a Dovecot server on Ubuntu 22.04, which works fine with Oauth2. I am 
building a new container based on Ubuntu 24.04. IMAP seems to work fine with 
plain authentication, but oauth2 fails (Dovecot v2.3.21). Same configuration as 
before.

However, I got this message:

May 09 09:41:57 auth: Error: 
oauth2(f...@mydomain.eu,10.110.41.32,): oauth2 failed: 
Introspection failed: No username returned

Weird, as it does indeed seems to get the username (as seen on the log). 

Any clue on why this fails? 

Best,

Francis
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Failing to archive many messages

[Francis Augusto Medeiros-Logeay via dovecot]

I noticed that my sieve rules are taking a very long time when archiving 
messages. I can see on my logs that every message takes about two-three seconds 
to be evaluated. 

My sieve for the Archive is this:

cat /var/mail/vmail/mail/sieve/global/report-ham.sieve
require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.mailbox" "*" {
  set "mailbox" "${1}";
}

if string "${mailbox}" "Trash" {
  stop;
}

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "sa-learn-ham.sh" [ "${username}" ];

And my sa-learn-ham.sh is this:

#!/bin/sh
export PATH="/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin:$PATH"
exec /usr/bin/sa-learn -u ${1} —ham




May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
multi-script: Start execute sequence
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
Executing script from `/var/mail/vmail/mail/sieve/global/report-ham.svbin'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
multi-script: Run script `/var/mail/vmail/mail/sieve/global/report-ham.svbin'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
Started running script `/var/mail/vmail/mail/sieve/global/report-ham.svbin'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
Finished running script `ok' 
(status=/var/mail/vmail/mail/sieve/global/report-ham.svbin, resource usage: no 
usage recorded)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
multi-script: Execute result
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Executing result (status=ok, commit=no)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Starting execution of actions
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Executing actions
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Executing pipe action
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
action pipe: running program: sa-learn-ham.sh
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Created
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Pass 
environment: USER=fran...@med-lo.eu
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Pass 
environment: HOME=/var/mail/vmail/med-lo.eu/francis
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Pass 
environment: HOST=fb58607074fd
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: Mailbox 
Archive: UID 191458: Opened mail because: mail stream
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Finished executing pipe action (status=ok, keep=implicit)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Finished executing actions (status=ok, keep=implicit)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Finished executing result (no commit, status=ok, keep=yes)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
multi-script: Sequence active
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
multi-script: Finishing sequence (status=ok)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Executing result (status=ok, commit=yes)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Starting execution of actions
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Executing actions
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Finished executing actions (status=ok, keep=implicit)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Execute implicit keep (failure=no)
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Start storing into mailbox Archive
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Executing implicit keep action
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Execute storing into mailbox 'Archive'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: Debug: sieve: 
uid=191458: Updated existing mail in mailbox 'Archive'
May 09 09:17:10 imap(fran...@med-lo.eu)<9349><4gm6NAAYsagKCige>: 

Re: Failing to archive many messages

[Francis Augusto Medeiros-Logeay via dovecot]

keep action
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: sieve: uid=25719: Execute storing into mailbox 'Archive'
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: sieve: uid=25719: Updated existing mail in mailbox 'Archive'
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: sieve: uid=25719: Finished executing implicit keep action 
(status=ok)
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: sieve: uid=25719: Finalizing actions
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: sieve: uid=25719: Finalize pipe action (status=ok, 
action_status=ok, commit_status=ok, pre-commit=yes)
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: sieve: uid=25719: Commit pipe action
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: 
Establishing connection
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh: Forked 
child process
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh (416): 
Connected to program
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh (416): 
Finished streaming payload to program
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh (416): 
Finished input to program
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh (416): 
Disconnected
May 01 14:20:04 imap(fran...@mydomain.com)<226>: 
Debug: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh (416): 
Waiting for program to finish after 6 msecs (timeout = 9994 msecs)



The client stays just like «Moving 1 of 4000 messages».

Then the logs start to go like this:

May 01 14:43:09 imap: Error: ERROR: the Bayes learn function returned an 
error, please re-run with -D for more information at /usr/bin/sa-learn 
line 500.
May 01 14:43:12 imap: Error: ERROR: the Bayes learn function returned an 
error, please re-run with -D for more information at /usr/bin/sa-learn 
line 500.

^C
root@fb58607074fd:/# tail -f /var/log/dovecot.log | grep Error
May 01 14:43:15 imap: Error: plugin: eval failed: interrupted at 
/usr/bin/sa-learn line 513.
May 01 14:43:15 imap: Error: ERROR: the Bayes learn function returned an 
error, please re-run with -D for more information at /usr/bin/sa-learn 
line 500.
May 01 14:43:15 imap(fran...@mydomain.com)<295>: 
Error: sieve: failed to pipe message to program `sa-learn-ham.sh': refer 
to server log for more information. [2024-05-01 14:43:15]
May 01 14:43:15 imap(fran...@mydomain.com)<295>: 
Error: sieve: Execution of script 
/var/mail/vmail/mail/sieve/global/report-ham.sieve failed
May 01 14:43:16 imap(fran...@mydomain.com)<292>: 
Error: program 
exec:/var/mail/vmail/mail/sieve/global/scripts/sa-learn-ham.sh (2978): 
Forcibly terminated with signal 15
May 01 14:43:16 imap(fran...@mydomain.com)<292>: 
Error: sieve: failed to pipe message to program `sa-learn-ham.sh': refer 
to server log for more information. [2024-05-01 14:43:16]
May 01 14:43:16 imap(fran...@mydomain.com)<292>: 
Error: sieve: Execution of script 
/var/mail/vmail/mail/sieve/global/report-ham.sieve failed
May 01 14:43:17 imap: Error: plugin: eval failed: interrupted at 
/usr/bin/sa-learn line 513.
May 01 14:43:17 imap: Error: ERROR: the Bayes learn function returned an 
error, please re-run with -D for more information at /usr/bin/sa-learn 
line 500.



For many minutes, the mail client gets an error about reaching the max 
connections from ip, and the logs go on and on with the errors above.


What could be the issue?

Best,
Francis
--
Francis Augusto Medeiros-Logeay
Oslo, Norway
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: ENV{"PATH"}. errors on my log

[Francis Augusto Medeiros-Logeay via dovecot]

Thanks!

I actually had a bash script that calls sa-learn, so I added `export 
PATH ...` there, and it worked.


Best,

Francis

---
Francis Augusto Medeiros-Logeay
Oslo, Norway

On 2024-05-01 13:23, Aki Tuomi wrote:

On 01/05/2024 13:59 EEST Francis Augusto Medeiros-Logeay via dovecot 
 wrote:


Hi,

I'm getting this error a lot on my logs:

May 01 12:57:03 imap: Error: Use of uninitialized value $ENV{"PATH"} 
in
split at /usr/lib/x86_64-linux-gnu/perl-base/File/Spec/Unix.pm line 
136.


I googled a bit and the only reference I found was related to the
possibility of imapsieve scripts that could be generating this. I
haven't found any perl script on my install, but I momentarily turned
off spamassassin, but the logs continued.

Any hint on how I can debug this? Turning the debug logs on dovecot
didn't bring me any extra information of what the source of this could
be.

Best,
Francis

--
Francis Augusto Medeiros-Logeay
Oslo, Norway
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Try adding `import_environment = $import_environment PATH` to your 
config.


Aki

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Failing to archive many messages

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

When I archive one or two messages, the messages are correctly moved to the 
Archive.

However, when I archive more messages - like 50 or more, dovecot seems to fail. 
The error I get is this one:

 Warning: Failed to do incremental sync for mailbox Archive, retry with a full 
sync (Modseq 17277 no longer in transaction log (highest=17323, 
last_common_uid=12216, nextuid=15767))

No messages end up being moved to the Archive folder. I tried to do this both 
from a mail client and from roundcube.

What could be the problem here? 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

ENV{"PATH"}. errors on my log

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

I'm getting this error a lot on my logs:

May 01 12:57:03 imap: Error: Use of uninitialized value $ENV{"PATH"} in 
split at /usr/lib/x86_64-linux-gnu/perl-base/File/Spec/Unix.pm line 136.


I googled a bit and the only reference I found was related to the 
possibility of imapsieve scripts that could be generating this. I 
haven't found any perl script on my install, but I momentarily turned 
off spamassassin, but the logs continued.


Any hint on how I can debug this? Turning the debug logs on dovecot 
didn't bring me any extra information of what the source of this could 
be.


Best,
Francis

--
Francis Augusto Medeiros-Logeay
Oslo, Norway
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Ouath2 compatible mail clients

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

Are there any oauth2 compatible mail clients out there? For Mac/iOS?

I see that Mail for Mac supports it, but just for Exchange. 
__
Francis Augusto Medeiros-Logeay
Oslo, Norway
Sent from a mobile device / Enviado a partir de dispositivo móvel
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [EXT] Re: dovecot and oauth2 (with keycloak) not working

[Francis Augusto Medeiros-Logeay via dovecot]

It seems that it works on the dovecot side. When Roundcube sends the 
token, I get the user authenticated via IMAP/oauth2 in dovecot.


What worked for me was:

- adding client_id and client_secret
- removing the tokeninfo_url, using just the introspect_url and 
introspect_mode=token.


Now gotta fix Roundcube...

Best,
Francis

On 2023-11-20 13:48, Francis Augusto Medeiros-Logeay wrote:

---
Francis Augusto Medeiros-Logeay
Oslo, Norway

On 2023-11-20 09:04, Aki Tuomi wrote:
On 20/11/2023 10:03 EET Francis Augusto Medeiros-Logeay 
 wrote:



> Try adding /?token= to tokeninfo_url. Dovecot 2.3.7.2 will simply
> concatenate tokeninfo_url and token, so you need to provide the URL in
> that fashion.
>
> Aki

Thanks Aki.

Still no go:


Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: 
Host

created




oauth2(fran...@mydomain.com,10.10.40.30,<4Gv83JAKyOcKCige>): oauth2
failed: Token validation failed: connect(10.10.200.10:443) failed:
Connection refused




___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [EXT] Re: dovecot and oauth2 (with keycloak) not working

[Francis Augusto Medeiros-Logeay via dovecot]

---
Francis Augusto Medeiros-Logeay
Oslo, Norway

On 2023-11-20 09:04, Aki Tuomi wrote:
On 20/11/2023 10:03 EET Francis Augusto Medeiros-Logeay 
 wrote:



> Try adding /?token= to tokeninfo_url. Dovecot 2.3.7.2 will simply
> concatenate tokeninfo_url and token, so you need to provide the URL in
> that fashion.
>
> Aki

Thanks Aki.

Still no go:


Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: Host
created




oauth2(fran...@mydomain.com,10.10.40.30,<4Gv83JAKyOcKCige>): oauth2
failed: Token validation failed: connect(10.10.200.10:443) failed:
Connection refused


It still doesn't work after I fixed that "connection refused" problem. I 
had a NAT reflector problem. But with that out of the way, I still don't 
get it to work:


Nov 20 13:43:03 auth: Error: 
ldap(fran...@mydomain.com,10.10.40.30,): ldap_bind() 
failed: Constraint violation
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: Host 
created
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: Host 
session created
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: Need 
to perform DNS lookup
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: 
Performing asynchronous DNS lookup
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client: dns(auth.mydomain.com): Lookup started
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client: Connecting
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=2800,uid=0): Client connected (fd=23)
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=2800,uid=0): Sending version handshake
Nov 20 13:43:03 auth: Debug: http-client[1]: request [Req1: GET 
https://auth.mydomain.com/realms/myrealm/protocol/openid-connect/userinfo?token=eyJhbGciOi..redacted...3MDA0ODQ0ODMsImlhdCI6MTcwMDQ4NDE4MywiYXV0aF...: 
Submitted (requests left=1)
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=2800,uid=0): dns(auth.mydomain.com): Lookup 
successful after 5 msecs
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: DNS 
lookup successful; got 1 IPs
Nov 20 13:43:03 auth: Debug: http-client: peer 10.10.20.100:443 
(shared): Peer created
Nov 20 13:43:03 auth: Debug: http-client: peer 10.10.20.100:443: Peer 
pool created
Nov 20 13:43:03 auth: Debug: http-client[1]: peer 10.10.20.100:443: Peer 
created
Nov 20 13:43:03 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Setting up connection to 10.10.20.100:443 
(SSL=auth.mydomain.com) (1 requests pending)
Nov 20 13:43:03 auth: Debug: http-client[1]: peer 10.10.20.100:443: 
Linked queue https://auth.mydomain.com:443 (1 queues linked)
Nov 20 13:43:03 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Started new connection to 
10.10.20.100:443 (SSL=auth.mydomain.com)
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=2800,uid=0): Disconnected: Connection closed 
(fd=23)
Nov 20 13:43:03 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=2800,uid=0): Disconnect: deinit
Nov 20 13:43:03 auth: Debug: http-client[1]: peer 10.10.20.100:443: 
Creating 1 new connections to handle requests (already 0 usable, 
connecting to 0, closing 0)
Nov 20 13:43:03 auth: Debug: http-client[1]: peer 10.10.20.100:443: 
Making new connection 1 of 1 (0 connections exist, 0 pending)
Nov 20 13:43:03 auth: Debug: http-client: conn 10.10.20.100:443 [1]: 
Connecting
Nov 20 13:43:03 auth: Debug: http-client: conn 10.10.20.100:443 [1]: 
Waiting for connect (fd=23) to finish for max 0 msecs
Nov 20 13:43:03 auth: Debug: http-client: conn 10.10.20.100:443 [1]: 
HTTPS connection created (1 parallel connections exist)
Nov 20 13:43:03 auth: Debug: http-client: conn 10.10.20.100:443 [1]: 
Client connected (fd=23)
Nov 20 13:43:03 auth: Debug: http-client: conn 10.10.20.100:443 [1]: 
Connected
Nov 20 13:43:03 auth: Debug: http-client: conn 10.10.20.100:443 [1]: 
Starting SSL handshake
Nov 20 13:43:03 auth: Debug: auth.mydomain.com: SSL: where=0x10, ret=1: 
before SSL initialization
Nov 20 13:43:03 auth: Debug: auth.mydomain.com: SSL: where=0x1001, 
ret=1: before SSL initialization
Nov 20 13:43:03 auth: Debug: auth.mydomain.com: SSL: where=0x1001, 
ret=1: SSLv3/TLS write client hello
Nov 20 13:43:03 auth: Debug: auth.mydomain.com: SSL: where=0x1002, 
ret=-1: SSLv3/TLS write client hello
Nov 20 13:43:03 auth: Debug: auth.mydomain.com: SSL: where=0x1002, 
ret=-1: SSLv3/TLS write client hello
Nov 20 13:43:03 auth: Debug: auth.mydomain.com: SSL: where=0x1002, 
ret=-1: SSLv3/TLS write client hello
Nov 20 13:43:03 auth: Debug: auth.mydomain.com: SSL: where=0x1001, 
ret=1: SSLv3/TLS write client hello
Nov 20 13:43:03 auth: Debug: auth.mydomain.com: SSL: where=0x1001, 
ret=1: SSLv3/TLS read server hello
Nov 20 13:43:03 auth: Debug: auth.

Re: dovecot and oauth2 (with keycloak) not working

[Francis Augusto Medeiros-Logeay via dovecot]

Try adding /?token= to tokeninfo_url. Dovecot 2.3.7.2 will simply 
concatenate tokeninfo_url and token, so you need to provide the URL in 
that fashion.


Aki


Thanks Aki.

Still no go:


Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: Host 
created
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: Host 
session created
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: Need 
to perform DNS lookup
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: 
Performing asynchronous DNS lookup
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client: dns(auth.mydomain.com): Lookup started
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client: Connecting
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=506,uid=0): Client connected (fd=24)
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=506,uid=0): Sending version handshake
Nov 20 08:59:19 auth: Debug: http-client[1]: request [Req1: GET 
https://auth.mydomain.com/realms/myrealm/protocol/openid-connect/userinfo/token=eyJhbGci...redacted...TcwMDQ2NzE1OSwiYXV0aF...: 
Submitted (requests left=1)
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=506,uid=0): dns(auth.mydomain.com): Lookup 
successful after 4 msecs
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: DNS 
lookup successful; got 1 IPs
Nov 20 08:59:19 auth: Debug: http-client: peer 10.10.200.10:443 
(shared): Peer created
Nov 20 08:59:19 auth: Debug: http-client: peer 10.10.200.10:443: Peer 
pool created
Nov 20 08:59:19 auth: Debug: http-client[1]: peer 10.10.200.10:443: Peer 
created
Nov 20 08:59:19 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Setting up connection to 10.10.200.10:443 
(SSL=auth.mydomain.com) (1 requests pending)
Nov 20 08:59:19 auth: Debug: http-client[1]: peer 10.10.200.10:443: 
Linked queue https://auth.mydomain.com:443 (1 queues linked)
Nov 20 08:59:19 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Started new connection to 
10.10.200.10:443 (SSL=auth.mydomain.com)
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=506,uid=0): Disconnected: Connection closed (fd=24)
Nov 20 08:59:19 auth: Debug: http-client: host auth.mydomain.com: conn 
unix:dns-client (pid=506,uid=0): Disconnect: deinit
Nov 20 08:59:19 auth: Debug: http-client[1]: peer 10.10.200.10:443: 
Creating 1 new connections to handle requests (already 0 usable, 
connecting to 0, closing 0)
Nov 20 08:59:19 auth: Debug: http-client[1]: peer 10.10.200.10:443: 
Making new connection 1 of 1 (0 connections exist, 0 pending)
Nov 20 08:59:19 auth: Debug: http-client: conn 10.10.200.10:443 [1]: 
Connecting
Nov 20 08:59:19 auth: Debug: http-client: conn 10.10.200.10:443 [1]: 
Waiting for connect (fd=24) to finish for max 0 msecs
Nov 20 08:59:19 auth: Debug: http-client: conn 10.10.200.10:443 [1]: 
HTTPS connection created (1 parallel connections exist)
Nov 20 08:59:19 auth: Debug: http-client: conn 10.10.200.10:443 [1]: 
Client connection failed (fd=24)
Nov 20 08:59:19 auth: Debug: http-client[1]: peer 10.10.200.10:443: 
Connection failed (1 connections exist, 0 pending)
Nov 20 08:59:19 auth: Debug: http-client: peer 10.10.200.10:443: Failed 
to make connection (1 connections exist, 0 pending)
Nov 20 08:59:19 auth: Debug: http-client[1]: peer 10.10.200.10:443: 
Failed to establish any connection within our peer pool: 
connect(10.10.200.10:443) failed: Connection refused (1 connections 
exist, 0 pending)
Nov 20 08:59:19 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Failed to set up connection to 
10.10.200.10:443 (SSL=auth.mydomain.com): connect(10.10.200.10:443) 
failed: Connection refused (1 peers pending, 1 requests pending)
Nov 20 08:59:19 auth: Debug: http-client[1]: peer 10.10.200.10:443: 
Unlinked queue https://auth.mydomain.com:443 (0 queues linked)
Nov 20 08:59:19 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Failed to set up any connection; failing 
all queued requests
Nov 20 08:59:19 auth: Debug: http-client[1]: request [Req1: GET 
https://auth.mydomain.com/realms/myrealm/protocol/openid-connect/userinfo/token=eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJaYTFXcXhxb0RULXBSc2o1WXZFdUJfLUxBVUtGNk5SeFFrUS1mNmdTUGs4In0.eyJleHAiOjE3MDA0Njc0NTksImlhdCI6MTcwMDQ2NzE1OSwiYXV0aF...: 
Error: 9003 connect(10.10.200.10:443) failed: Connection refused
Nov 20 08:59:19 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Dropping request [Req1: GET 

dovecot and oauth2 (with keycloak) not working

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

I successfully configured Roundcube to use keycloak for oauth2.

However, I am having trouble to make it work with dovecot. My configuration is 
this:

cat dovecot-oauth2.conf.ext
tokeninfo_url = 
https://auth.mydomain.com/realms/myrealm/protocol/openid-connect/userinfo
introspection_url = 
https://auth.mydomain.com/realms/myrealm/protocol/openid-connect/token/introspect
introspection_mode = post
username_attribute = postfixMailAddress
debug = yes
scope = openid Roundcube_email

This is what I am getting from the logs:


Nov 20 08:20:30 auth: Error: 
ldap(fran...@mydomain.com,10.10.40.30,): ldap_bind() failed: 
Constraint violation
Nov 20 08:20:30 auth: Debug: http-client: host auth.mydomain.com: Host created
Nov 20 08:20:30 auth: Debug: http-client: host auth.mydomain.com: Host session 
created
Nov 20 08:20:30 auth: Debug: http-client: host auth.mydomain.com: IPs have 
expired; need to refresh DNS lookup
Nov 20 08:20:30 auth: Debug: http-client: host auth.mydomain.com: Performing 
asynchronous DNS lookup
Nov 20 08:20:30 auth: Debug: http-client[1]: request [Req1: GET 
https://auth.mydomain.com/realms/med-lo/protocol/openid-connect/userinfoeyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJaYTFXcXhxb0RULXBSc2o1WXZFdUJfLUxBVUtGNk5SeFFrUS1mNmdTUGs4In0.eyJleHAiOjE3MDA0NjUxMzAsImlhdCI6MTcwMDQ2NDgzMCwiYXV0aF90aW1lIjoxNzAwNDY0Njg5LCJqdGkiOiIzZTk5YWI4Yi0xZTkyLTRlMDYtYjg0NC1kODc4ZDZjODZjOWMiLCJpc3MiOiJodHRwczovL2F1dGgubWVkLWxvLmV1L3JlYWxtcy9tZWQtbG8iLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiZGE5MDk4NDQtNjlmOS00ZjkzLWI1NjctMGZjOWQ3YzA3MTZmIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoicm91bmRjdWJlIiwic2Vzc2lvbl9zdGF0ZSI6ImZkY2I2YTczLTNjNjgtNDlhNS1hOTlkLTdkYmE4ODNlNjg4NiIsImFjciI6IjAiLCJhbGxvd2VkLW9yaWdpbnMiOlsiaHR0cHM6Ly9tYWlsLm1lZC1sby5ldSJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsImRlZmF1bHQtcm9sZXMtbWVkLWxvIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgUm91bmRjdWJlX2VtYWlsIHByb2ZpbGUgZW1haWwgb3BlbmVkIiwic2lkIjoiZmRjYjZhNzMtM2M2OC00OWE1LWE5OWQtN2RiYTg4M2U2ODg2IiwiZW1haWxfdmVyaWZpZWQiOnRydWUsInBvc3RmaXhNYWlsQWRkcmVzcyI6ImZyYW5jaXNAbWVkLWxvLmV1IiwicG9zdGZpeE1haWxib3giOiJmcmFuY2lzQG1lZC1sby5ldSIsIm5hbWUiOiJGcmFuY2lzIEF1Z3VzdG8gTWVkZWlyb3MtTG9nZWF5IiwicHJlZmVycmVkX3VzZXJuYW1lIjoiZnJhbmNpcyIsImdpdmVuX25hbWUiOiJGcmFuY2lzIEF1Z3VzdG8iLCJmYW1pbHlfbmFtZSI6Ik1lZGVpcm9zLUxvZ2VheSIsImVtYWlsIjoiZnJhbmNpc0BtZWQtbG8uZXUifQ.Cehd8sbCTihfq1SKQitLTPfZZAWHx31sy8I6YydY_3eZvyHRellhQz1F9NxFt0uHaFk3KeddHV6U9z14qT7fStDp18ECJodSdcDt4k6J7geNjSbO3jSXOfk5JTbNPv0agi9e767E54g2ZkStPEezrAYY83msx7JSVpEmwKItSrDyyAWH44jp0OsnaLVCOZP1gBklTgiDt7uVsFwL9kpGamsMt62jNADnIAt6qLapHofiXi7GuIKdQP8-IG_7cCcpY6bEvcHiSgqhIpk5UHgMsljNQOkCKDpQ5rrTmRxloVF1y1zE7LYPNcugC_ZF_5TzxhVTEdEOLL9Q5epdlJvtvQ]:
 Submitted (requests left=1)
Nov 20 08:20:30 auth: Debug: http-client: host auth.mydomain.com: DNS lookup 
successful; got 1 IPs
Nov 20 08:20:30 auth: Debug: http-client: peer 10.10.100.10:443 (shared): Peer 
created
Nov 20 08:20:30 auth: Debug: http-client: peer 10.10.100.10:443: Peer pool 
created
Nov 20 08:20:30 auth: Debug: http-client[1]: peer 10.10.100.10:443: Peer created
Nov 20 08:20:30 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Setting up connection to 10.10.100.10:443 
(SSL=auth.mydomain.com) (1 requests pending)
Nov 20 08:20:30 auth: Debug: http-client[1]: peer 10.10.100.10:443: Linked 
queue https://auth.mydomain.com:443 (1 queues linked)
Nov 20 08:20:30 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Started new connection to 10.10.100.10:443 
(SSL=auth.mydomain.com)
Nov 20 08:20:30 auth: Debug: http-client[1]: peer 10.10.100.10:443: Creating 1 
new connections to handle requests (already 0 usable, connecting to 0, closing 
0)
Nov 20 08:20:30 auth: Debug: http-client[1]: peer 10.10.100.10:443: Making new 
connection 1 of 1 (0 connections exist, 0 pending)
Nov 20 08:20:30 auth: Debug: http-client: conn 10.10.100.10:443 [1]: Connecting
Nov 20 08:20:30 auth: Debug: http-client: conn 10.10.100.10:443 [1]: Waiting 
for connect (fd=23) to finish for max 0 msecs
Nov 20 08:20:30 auth: Debug: http-client: conn 10.10.100.10:443 [1]: HTTPS 
connection created (1 parallel connections exist)
Nov 20 08:20:30 auth: Debug: http-client: conn 10.10.100.10:443 [1]: Client 
connection failed (fd=23)
Nov 20 08:20:30 auth: Debug: http-client[1]: peer 10.10.100.10:443: Connection 
failed (1 connections exist, 0 pending)
Nov 20 08:20:30 auth: Debug: http-client: peer 10.10.100.10:443: Failed to make 
connection (1 connections exist, 0 pending)
Nov 20 08:20:30 auth: Debug: http-client[1]: peer 10.10.100.10:443: Failed to 
establish any connection within our peer pool: connect(10.10.100.10:443) 
failed: Connection refused (1 connections exist, 0 pending)
Nov 20 08:20:30 auth: Debug: http-client[1]: queue 
https://auth.mydomain.com:443: Failed to set up connection to 10.10.100.10:443 
(SSL=auth.mydomain.com): 

Shared mailboxes and indexes

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

I googled a bit to see if I understand it, but no, I haven’t got anything that 
explains this:

What’s exactly the difference between INDEX and INDEXPVT for shared mailboxes? 
Why are they pointing to the same folder on Dovecot example 
(https://doc.dovecot.org/configuration_manual/shared_mailboxes/shared_mailboxes/#user-shared-mailboxes)?

Best,

Francis
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Changed separator, sub folders not showing new e-mail

[Francis Augusto Medeiros-Logeay via dovecot]

I found the answer here: 
https://apple.stackexchange.com/questions/441313/no-mailbox-doesnt-exist-separator-woes-in-mail-app-dovecot-interaction/441341#441341

Apparently on Mac one needs to update the separator on a file, since it doesn’t 
seem that the Mail.app does it for itself.

On the iPhone it happened automatically.

Best,

Francis 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Changed separator, sub folders not showing new e-mail

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

I recently changed the mailbox separator from '.' to '/' so that I could 
have shared mailboxes working.


However, while the mailboxes show fine on Roundcube and on other 
web-based e-mail clients, on the Mail.app on my Mac I don't get newer 
messages on those sub folders.


For example, I have a subfolder Test. Mails filtered to be dropped 
there, show fine.
But I have one called Lists, with a sub folder called "Dovecot". I can't 
read mails that are under "Lists/Dovecot".


Can anyone help?

Best,

Francis

--
Francis Augusto Medeiros-Logeay
Oslo, Norway
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Separators and shared namespaces

[Francis Augusto Medeiros-Logeay via dovecot]

Hi Martin,

I had a similar problem:

https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/DNHM3OOV3SZUC2SIS7FMWL7MMT64F73T/


It seemed to me that using ‘.’ truncated the domain name when that was used in 
the paths, such as when using %d.

Best,

Francis 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Dovecot lda-dupes

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

I have read elsewhere that the Maildir shouldn’t be at the home. But that’s how 
it worked for me since the start.

I wonder if I could either change the location of these .dovecot.lda-dupes, or 
if I could change home without having to move the mailboxes of all the users.

Today I have this:

On my dovecot-sql.conf.ext:

iterate_query = SELECT username AS user FROM mailbox
user_query = SELECT CONCAT('/var/mail/vmail/',maildir) AS home, \
  
CONCAT('maildir:/var/mail/vmail/',maildir,':INDEX=/var/vmail-index/index/%d/%n:CONTROL=/var/vmail-index/control/%d/%n')
 AS mail, \
  5000 AS uid, 5000 AS gid, CONCAT('*:bytes=',quota) AS quota_rule \
  FROM mailbox WHERE username = '%u' AND active = 1
password_query = SELECT username AS user,password FROM mailbox \
  WHERE username = '%u' AND active=‘1'

On my 10-mail.conf

mail_location = 
maildir:/var/mail/vmail/%d/%n:INDEX=/var/vmail-index/index/%d/%n:CONTROL=/var/vmail-index/control/%d/%n

If I add `Maildir` to the mail_location and to the mail user_query, will I have 
to move mailboxes, or the only thing I have to do is to add “Maildir” when I 
use %h/%%h? 

Best,

Francis 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Dovecot lda-dupes

[Francis Augusto Medeiros-Logeay via dovecot]

Never mind, I found it. I had to add 

`replication_dsync_parameters`, with the default values, but with `-x shared` 
to exclude the namespace.

Best,

Francis 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Dovecot lda-dupes

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

Since I enabled shared mailboxes I'm having these errors:

Sep 27 21:22:41 doveadm(fran...@med-lo.eu): Error: Failed to access 
mailbox dovecot/lda-dupes: 
stat(/var/mail/vmail/med-lo.eu/francis/.dovecot.lda-dupes/tmp) failed: 
Not a directory


And replication stopped.

How can I fix this?

Best,
Francis
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Separators and shared namespaces

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

When I used a ‘.’  as a separator, it seemed to truncate paths with the domain 
name, such as those with %%d. That happened only to shared namespaces.

Best,

Francis 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Exclude shared mailboxes from replication

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

I was wondering if it is possible to exclude shared mailboxes from replication.

I have us...@mydomain.com sharing mailboxes to us...@mydomain.com. 

However, since both mailboxes already replicate to a backup server, I wonder if 
it is possible to exclude the replicated mailboxes  from syncing.

Best,

Francis 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: All list=yes namespaces must use the same separator - don't they?

[Francis Augusto Medeiros-Logeay via dovecot]

It works everywhere now. I guess it just needed its sweet time.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: All list=yes namespaces must use the same separator - don't they?

[Francis Augusto Medeiros-Logeay via dovecot]

I found what the problem was. It seems that using ‘.’ as a separator worked. 
But somehow folders and domain names with dots got messed up:

maildir++: root=/var/mail/vmail/francisaugusto/francis, 
index=/var/mail/vmail/med-lo.eu/francis/shared/index/francis@francisaugusto, 
indexpvt=/var/mail/vmail/med-lo.eu/francis/shared/francis@francisaugusto, 
control=, inbox=/var/mail/vmail/francisaugusto/francis, alt=
Sep 25 19:10:57 imap(fran...@med-lo.eu)<58061>: Debug: Namespace 
shared.francis@francisaugusto.: Creating storage despite: Root mail directory 
doesn't exist: /var/mail/vmail/francisaugusto/francis

You see, my domain is francisaugusto.com , but it 
got truncated. 

I changed the separator to ‘/‘ on the Inbox namespace and on the shared one, 
and voila, it worked.

I can see my shared mailboxes everywhere. But on Apple Mail (both on the Mac 
and on the iPhone), the shared mailboxes show no message. 

Any clue why? 
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

All list=yes namespaces must use the same separator - don't they?

[Francis Augusto Medeiros-Logeay via dovecot]

Hi,

I am trying to configure shared mailboxes, so I enabled the acl and the 
imap_acl plugins.


But I got the "All list=yes namespaces must use the same separator" 
error when I added this:


namespace {
  type = shared
   #separator =

  # Mailboxes are visible under "shared/user@domain/"
  # %%n, %%d and %%u are expanded to the destination user.
   prefix = shared/%%u/

  # Mail location for other users' mailboxes. Note that %variables and 
~/
  # expands to the logged in user's data. %%n, %%d, %%u and %%h expand 
to the

  # destination user's data.
  location = 
maildir:/var/mail/vmail/%%d/%%n:INDEX=/var/mail/vmail-index/index/%%d/%%n:INDEXPVT=/var/mail/vmail-index/index/%%d/%%n


  # Use the default namespace for saving subscriptions.
  subscriptions = no

  # List the shared/ namespace only if there are visible shared 
mailboxes.

  list = children
}

I don't have any separator configured explicitly, and when I see the 
config, the value is empty. So I don't really know what is the problem 
here.


Any hints?

Best,
Francis

--
Francis Augusto Medeiros-Logeay
Oslo, Norway
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: LDAP just for passdb

[Francis Augusto Medeiros-Logeay]

Again, a bit more reading got me to adding this to my passdb config:

  username_filter = *@domain-a.com

This way, I can control which domains get to authenticate via my ldap backend, 
which gives me time to design a good way of saving other attributes there.

If anyone have other ways of doing this, ie., having multiple domains on 
ldap/freeipa and getting an elegant integration with Dovecot, I’d be glad to 
hear.

Best,

Francis

> On 14 Oct 2022, at 21:58, dovecot-requ...@dovecot.org wrote:
> 
> I actually saw that it was possible, and it works, but I came across another 
> problem and I wonder if you have any tips about it:
> 
> On my current dovecot setup, I use SQL as the backend. So I have the 
> following users:
> 
> fran...@domain-a.com  
> 
> fran...@domain-b.com  
> 
> 
> Those are separate users which their own mailboxes.
> 
> However, I have a freeipa that is configured for the `domain-a.com 
>  ` realm. However, since I am 
> using `%n` for the uid search:
> 
> auth_bind_userdn = uid=%n,cn=users,cn=accounts,dc=domain-a,dc=com
> And 
> pass_filter = (&(objectClass=posixAccount)(uid=%n))
> 
> It of course leads up to both users above being able to authenticate with the 
> same password.
> 
> Is there a way to limit ldap authentication to just one domain, or perform a 
> search where both username and domain are checked? I could use the 
> `mail``attribute to filter users, but I imagine that if two users have the 
> same mail configured, I?d run into trouble?.
> 
> Best,
> 
> Francis
> 
>> On 14 Oct 2022, at 20:08, dovecot-requ...@dovecot.org 
>>  wrote:
>> 
>> Hi,
>> 
>> I couldn't find it in the documentation, so I was wondering - is it 
>> possible to configure Dovecot to use LDAP for passdb and keep using SQL 
>> for userdb?
>> 
>> I would like to do that before I come up with a good strategy to expand 
>> my ldap schema to support other mail attributes for virtual domains, 
>> aliases, etc.
>> 
>> I am currently using FreeIPA.
>> 
>> Best,
>> 
>> Francis
> 

Re: LDAP just for passdb

[Francis Augusto Medeiros-Logeay]

I actually saw that it was possible, and it works, but I came across another 
problem and I wonder if you have any tips about it:

On my current dovecot setup, I use SQL as the backend. So I have the following 
users:

fran...@domain-a.com 
fran...@domain-b.com 

Those are separate users which their own mailboxes.

However, I have a freeipa that is configured for the `domain-a.com 
` realm. However, since I am using `%n` for the uid 
search:

auth_bind_userdn = uid=%n,cn=users,cn=accounts,dc=domain-a,dc=com
And 
pass_filter = (&(objectClass=posixAccount)(uid=%n))

It of course leads up to both users above being able to authenticate with the 
same password.

Is there a way to limit ldap authentication to just one domain, or perform a 
search where both username and domain are checked? I could use the 
`mail``attribute to filter users, but I imagine that if two users have the same 
mail configured, I’d run into trouble….
 
Best,

Francis

> On 14 Oct 2022, at 20:08, dovecot-requ...@dovecot.org wrote:
> 
> Hi,
> 
> I couldn't find it in the documentation, so I was wondering - is it 
> possible to configure Dovecot to use LDAP for passdb and keep using SQL 
> for userdb?
> 
> I would like to do that before I come up with a good strategy to expand 
> my ldap schema to support other mail attributes for virtual domains, 
> aliases, etc.
> 
> I am currently using FreeIPA.
> 
> Best,
> 
> Francis

LDAP just for passdb

[Francis Augusto Medeiros-Logeay]

Hi,

I couldn't find it in the documentation, so I was wondering - is it 
possible to configure Dovecot to use LDAP for passdb and keep using SQL 
for userdb?


I would like to do that before I come up with a good strategy to expand 
my ldap schema to support other mail attributes for virtual domains, 
aliases, etc.


I am currently using FreeIPA.

Best,

Francis

Dovecot, sa-learn and sieve - where to save the Bayes DB

[Francis Augusto Medeiros-Logeay]

Hi!

I have dovecot, Spamassassin and postfix running. I started to use 
sa-learn to get better and more precise spam identification and 
filtering.


I have not set the Bayes_path preference on my local.cf file, thus when 
I call sa-learn -u myu...@mydomain.org --spam, the Bayes DB gets updated 
at /root/.spamassassin.


I decided then to create a sieve filter on dovecot, as described on 
Dovecot's wiki page. Basically, Dovecot's sieve plugin execute sa-learn 
with a -u argument, apparently the same way I do when I use it manually.


BUT, the thing is that when Dovecot executes sa-learn, the bayes 
database is created on the virtual user's folder, ie., 
/var/mail/vmail/mydomain.org/user/.spamassassin. When I call it 
manually, the db used is /root/.spamassassin`


I am now confused as I don't know if, when evaluating spam, spamd will 
use the Bayes DB on the user's Maildir, or if it will use the one under 
/root/.spamassassin.


I would really like to have my spam evaluation done per user, not site 
wide, so I wonder:


- How can I configure bayes_path for each user, individually, on a 
virtual users setup?
- Does calling spamd with --virtual-config-dir have anything to do with 
this? I tried it, setting it up with my virtual users path, but it 
didn't seem to make any difference when it comes to what I describe in 
this post. With or without it set up, the results above are the same.


--
Francis Augusto Medeiros-Logeay
Oslo, Norway

Re: Trying to use solr

[Francis Augusto Medeiros-Logeay]

On 26.07.2020 22:45, Shawn Heisey wrote:

On 7/26/2020 3:48 AM, r...@med-lo.eu wrote:

Thanks a lot Shawn. I will send the full error when I get home a week
from now. But this was my first attempt to use solr - is it still
subject to those conflicts?


You'll have to be more specific ... but I believe the answer to your
question is likely to be "yes.

I am the person who wrote the "HowToReindex" wiki page that I linked.
Nothing in that wiki page is addressed at any particular version of
Solr.  The possibility of a schema change requiring a complete reindex
(and sometimes deleting the index directory entirely) applies to ANY
version.



Ok, I am trying again. I get no errors when doing a `doveadm fts 
rescan`, but get errors when trying this:


doveadm index -u myu...@mydomain.com INBOX
doveadm(myu...@mydomain.com): Error: fts_solr: Indexing failed: 400 Bad 
Request
doveadm(myu...@mydomain.com): Error: Mailbox INBOX: Transaction commit 
failed: FTS transaction commit failed: backend deinit


I guess this is a matter of waiting the reindex to be over?

I get so many "Type mismatch" errors in Solr, except for this one that 
looks different and showed up after trying the doveadm index command 
above:



ERROR true
x:dovecot
RequestHandlerBase
org.apache.solr.common.SolrException: Exception writing document id 
210/9fd7941e8297d25d9160c3fdd3da/myu...@mydomain.com to the index; 
possible analysis error: cannot change field "box" from index 
options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS
org.apache.solr.common.SolrException: Exception writing document id 
210/9fd7941e8297d25d9160c3fdd3da/myu...@mydomain.com to the index; 
possible analysis error: cannot change field "box" from index 
options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS
	at 
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:246)
	at 
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76)
	at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
	at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:259)
	at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:489)
	at 
org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:339)
	at 
org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50)
	at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:339)
	at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:225)
	at 
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
	at 
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261)

at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188)
	at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
	at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
	at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:198)

at org.apache.solr.core.SolrCore.execute(SolrCore.java:2576)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:803)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:582)
	at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:424)
	at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
	at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
	at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
	at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
	at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
	at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
	at 

Re: Doveadm Backup issues

[Francis Augusto Medeiros-Logeay]

On 6 Jun 2020, at 01:10, Bernardo Reino  wrote:
> 
> On Fri, 5 Jun 2020, Francis Augusto Medeiros-Logeay wrote:
> 
>> Another related question: if backing up on another location on the same 
>> computer, using -A doesn't seem to recreate the structure of the original 
>> Maildir (for example, /var/mail/%d%n).
>> 
>> Is there a way to do that?
> 
> If I understand your question correctly (I may be missing context), you can 
> use dsync for that, e.g.:
> 
> DEST=mdbox:/path/to/$MAILBOX/mdbox
> dsync -u $MAILBOX backup $DEST
> 
> where DEST is in "mail location" format (could also be 
> "maildir:/path/to/Maildir")
> and $MAILBOX is a dovecot (virtual) user.
> 
> My backup script does this:
> 
> USERS=$(cat /etc/dovecot/virtual_passwd | cut -d: -f1)
> for MAILBOX in $USERS; do
>DEST=mdbox:/encrypted/snap_mail/$MAILBOX/mdbox
>dsync -v -u $MAILBOX backup $DEST
> done
> 
> This runs periodically and does the local "backup".
> Another server picks that up, also periodically, using rsync, which is the 
> real backup.
> 
> Cheers.

Thanks Bernardo. That’s what I want to do. But it’s just that it is confusing 
when there is a -A parameter that backups up all users, but apparently no way 
to specify individual paths on the destination. So I guess the -A option is 
only useful when backing up to a remote location that already has the same set 
of users. 

It would be nice to be able to soecify a format for the destination path 
without having use loops. But loop it is, then. :)

Best, 

Francis 

Re: Doveadm Backup issues

[Francis Augusto Medeiros-Logeay]

Hi,

As said below, I am trying to use doveadm backup. I made some progress, 
but I think I misunderstand some of the basic principles behind the 
command.


My main issue is: when using the -A flag, is there a way to create the 
backup user on the remote server if it doesn't exist there?


Another related question: if backing up on another location on the same 
computer, using -A doesn't seem to recreate the structure of the 
original Maildir (for example, /var/mail/%d%n).


Is there a way to do that?

Best,

Francis

On 02.06.2020 14:31, Francis Augusto Medeiros-Logeay wrote:

Hi,

I'm trying to use doveadm backup but I'm having a few issues.

When trying locally:

`doveadm backup -D -A maildir:/usr/local/backup`

It results that no mailbox is separated by user - all seems to sync on
the same folder, messing up with the folder structure of the original
maildir.

When trying remotely:

`doveadm backup -D -A -N ssh -i mykey r...@myremotebackup.com:/backup
doveadmin dsync-server -A`

I get this error line for each user:

dsync-local(theu...@thedomain.com)<5DNrEXdG1l5tAgAAqsACHw>: Error:
read(remote) failed: EOF (version not received)
doveadm(theu...@thedomain.com): Fatal: execvp(ssh) failed: No such
file or directory

I'd be very thankful if someone could help me with this.

Best,

Francis

Doveadm Backup issues

[Francis Augusto Medeiros-Logeay]

Hi,

I'm trying to use doveadm backup but I'm having a few issues.

When trying locally:

`doveadm backup -D -A maildir:/usr/local/backup`

It results that no mailbox is separated by user - all seems to sync on 
the same folder, messing up with the folder structure of the original 
maildir.


When trying remotely:

`doveadm backup -D -A -N ssh -i mykey r...@myremotebackup.com:/backup 
doveadmin dsync-server -A`


I get this error line for each user:

dsync-local(theu...@thedomain.com)<5DNrEXdG1l5tAgAAqsACHw>: Error: 
read(remote) failed: EOF (version not received)
doveadm(theu...@thedomain.com): Fatal: execvp(ssh) failed: No such file 
or directory


I'd be very thankful if someone could help me with this.

Best,

Francis

--
Francis Augusto Medeiros-Logeay
Oslo, Norway

Re: Simple backup of maildir folder

[Francis Augusto Medeiros-Logeay]

> On 31 May 2020, at 11:13, Laura Smith  
> wrote:
> 
> 
> 
> 
> Sent with ProtonMail Secure Email.
> 
> ‐‐‐ Original Message ‐‐‐
>> On Sunday, 31 May 2020 09:35, @lbutlr  wrote:
>> 
>> 
>> 
>> A couple of notes on this quite useful script:
>> 
>> My mktemp does not support -p (FreeBSD 12.1) is I had to change the script 
>> to:
>> 
> 
> 
> In my scripts I tend to create a tempdir and then tempfiles within that.  It 
> makes the cleanup routine neater, e.g. at the top of my scripts :
> 
> TEMP_DIR=$(mktemp -qd || { doLog "Failed to make temp dir !"; exit 1; })
> rmTmpFiles() { rm -rf "${TEMP_DIR}"; }
> createTempFile() { local MYTEMP=$(mktemp -qp "${TEMP_DIR}" || doLog "Failed 
> to create temp file"; exit 1); echo $MYTEMP; }
> 

I don’t think I need to clean up, and my “temp_dir” will end up being 
permanent, as to make doveadm backup snappier I will simply maintain the 
directory there and use crown jobs to update the dir. 


> Also my backup scripts have locking procedures built-in so as to avoid race 
> conditions.

Also not sure if that’s needed when using doveadm backup, as it takes care of 
the potential race conditions. 

So the idea is to use doveadm to a local folder, then rsync it to a remote 
server where snapshots can be easily created. 

It annoys me have to do the doveadm to the local server - seems like a waste of 
disk space, but since maildirlock is not working and since doveadm backup to a 
remote server requires a bit more work to, well, to work, so I guess this is a 
simple and quick solution. 

Best,

Francis 

Re: Simple backup of maildir folder

[Francis Augusto Medeiros-Logeay]

On 2020-05-30 07:49, Admin Beckspaced wrote:

Am 29.05.2020 um 17:37 schrieb Francis Augusto Medeiros-Logeay:

Hi,

I want to do a simple backup of my maildir. I read here that 
maildirlock is not to count on in the future and the recommended 
option was to use doveadm backup. However, it seems that this requires 
to setup another dovecot server.


My questions are:

- How much to I need to configure this new server? Do I need to setup 
user by user, or even install postfix (that's where my users are 
configured) to get it all backed up? I tried to install dovecot and 
running doveadm backup, but it complains the users are not there.
- What's the simplest way to configure dovecot without mirrowing my 
server so that this can work?


Best,

Francis

---
Francis Augusto Medeiros-Logeay
Oslo, Norway


Hello Francis,

I also have maildir as mailbox format and use the following script to
do daily backups

https://github.com/tachtler/dovecot-backup

There's also a more detailed how-to which explains how to do a restore.
But it's only in german language

https://www.dokuwiki.tachtler.net/doku.php?id=tachtler:dovecot_backup_-_skript

Google Translate is your friend ;)
perhaps this might help?

greetings
Becki



Thanks a lot Becki. I read your script, but the thing is that I want to 
backup to a remote server. Your script seems to be focused on a local 
backup. Did I get it right? But it still gives me a good idea of a 
possibility of backing up locally and rsyncing to another server.


--
Francis Augusto Medeiros-Logeay
Oslo, Norway

Re: migrating dovecot to new server

[Francis Augusto Medeiros-Logeay]

I did what Aki said. Actually I moved it all to a new server and to docker 
containers. From Dovecot 2.2 to 2.3. And it worked fine. Of course one always 
forget something - the path for the certificates, the different directory for 
mounting storage, etc. 

But am surprise and happy that it worked very well. 

__
Francis Augusto Medeiros-Logeay
Oslo, Norway
Sent from a mobile device / Enviado a partir de dispositivo móvel

> On 29 May 2020, at 19:33, Aki Tuomi  wrote:
> 
> 
>> On 29/05/2020 20:12 Ralph Seichter  wrote:
>> 
>> 
>> * David Mehler:
>> 
>>> I'd ideally like to back up all the mails in the maildir location,
>>> copy that over, load in my configuration files, and bring the system
>>> back up.
>> 
>> I don't think this qualifies as "ideal" for migration. My recommendation
>> is to set up the new server and configure Dovecot replication (see Wiki).
>> 
>> If you configured things correctly, migration will happen without
>> further intervention, and if there is a problem, you can resolve it in
>> your own time, without damaging the existing server and data.
>> 
>> -Ralph
> 
> If your directory structure *and* namespace structure stays the same, you can 
> rsync files over.
> 
> If you are going to change anything, do as Ralph suggests.
> 
> Aki

Simple backup of maildir folder

[Francis Augusto Medeiros-Logeay]

Hi,

I want to do a simple backup of my maildir. I read here that maildirlock 
is not to count on in the future and the recommended option was to use 
doveadm backup. However, it seems that this requires to setup another 
dovecot server.


My questions are:

- How much to I need to configure this new server? Do I need to setup 
user by user, or even install postfix (that's where my users are 
configured) to get it all backed up? I tried to install dovecot and 
running doveadm backup, but it complains the users are not there.
- What's the simplest way to configure dovecot without mirrowing my 
server so that this can work?


Best,

Francis

---
Francis Augusto Medeiros-Logeay
Oslo, Norway

Problems during replication

[Francis Augusto Medeiros-Logeay]

Hi,

I migrated my dovecot instance to a new server. I moved the mail 
directories, databases, etc.


Replication is on, and I keep having these errors:

May 02 16:19:45 dsync-local(fran...@francisaugusto.com): Error: Couldn't 
lock /var/mail/vmail/francisaugusto.com/francis//.dovecot-sync.lock: 
fcntl(/var/mail/vmail/francisaugusto.com/francis//.dovecot-sync.lock, 
write-lock, F_SETLKW) locking failed: Timed out after 30 seconds (WRITE 
lock held by pid 178)
May 02 16:19:45 doveadm: Error: Couldn't lock 
/var/mail/vmail/medeiros.co/inis//.dovecot-sync.lock: 
fcntl(/var/mail/vmail/medeiros.co/inis//.dovecot-sync.lock, write-lock, 
F_SETLKW) locking failed: Timed out after 30 seconds (WRITE lock held by 
pid 185)


See the errors are both from dsync-local and doveadm.

Any tips on how to fix this?

Best,

Francis

Replicating to an older version

[Francis Augusto Medeiros-Logeay]

Hi everyone,

I have two servers running dovecot, both at version 2.2.33.2. One is a 
an mx-backup and they replicate to each other.


I am moving the main server to a new VPS instance, and I'm planning the 
move carefully, including running dovecot on a container (Docker).


I am basing my container on Ubuntu 20.04, and the dovecot that installs 
is the 2.3.7.2.


My question is: will replication work ok once configured? Reading the 
documentation for version upgrade there was nothing on this. I will 
eventually upgrade the "slave" server, but it might take a few weeks.


Any tips on this would be greatly appreciated.

Best,

Francis

Replicating to an older version

[Francis Augusto Medeiros-Logeay]

Hi everyone,

I have two servers running dovecot, both at version 2.2.33.2. One is a 
an mx-backup and they replicate to each other.


I am moving the main server to a new VPS instance, and I'm planning the 
move carefully, including running dovecot on a container (Docker).


I am basing my container on Ubuntu 20.04, and the dovecot that installs 
is the 2.3.7.2.


My question is: will replication work ok once configured? Reading the 
documentation for version upgrade there was nothing on this. I will 
eventually upgrade the "slave" server, but it might take a few weeks.


Any tips on this would be greatly appreciated.

Best,

Francis

Re: Strategy for fts

[Francis Augusto Medeiros-Logeay]

This is very good news. I will certainly try it!

Thanks for that!

Best,

Francis

---
Francis Augusto Medeiros-Logeay
Oslo, Norway

On 15.02.2020 19:54, Joan Moreau wrote:

I updated fts-xapian to make it compatible with dovecot 2.2

On 2020-02-04 12:37, Peter Chiochetti wrote:


Am 04.02.20 um 11:46 schrieb Francis Augusto Medeiros-Logeay:


Hi Philon,

Thanks a lot for your thoughts!

Can I ask you if using Solr improved things for you? I have a
mailbox with 15 years of e-mail and searching things take a long
time.


Here, SOLR itself searches a quarter million mails in split seconds
and returns very good results. That is on a low memory average
machine.

If you dont mind the standard, you can change the schema, so headers
(from, to) get indexed in body text. That can help narrowing
results.

Only problem is search through e.g. nested folders from IMAP:
something like ESEARCH would be nice -
https://tools.ietf.org/html/rfc6237

Peter

On 04.02.2020 09:39, Philon wrote: Hi Francis,

next to fts-solr there was fts-lucene. But that Lucene there seems
heavily outdated why the Dovecot docs also suggest using Solr.
Elasticsearch probably is similar to Solr but the later is
maintained
by Dovecot team.

I started with downloading the Solr binary distribution to Debian
with
JRE preinstalled and things were running like after 10 min. Yes it's
a
bit more complicated to find the schema and edit things like header
size (in tips section). It's running quite nicely since then and has
zero maintenance.
I will try again - I kept getting some weird errors, so I don't know
if that's why I wasn't seing much of improvement.

As FTS indexes are separate in external Solr instance I'd guess that
it won't interfere with dsync. What I don't know is if dsync'ing
would
trigger indexing. This brings me to wonder how one could actually
replicate the Solr instance!?
Good question. But what I thought about doing was to install FTS on
my backup instance, and if things go fine, then I install an FTS
instance on my production server - that is, if one doesn't interfere
with the other.

I will give Solr another shot - my worries are mostly if Solr is
supported on ARM (my prod instance is running on ARM) - I know
Elasticsearch has an ARM build.

Ii thought about the Xapian engine, but since it requires dovecot
2.3, I will have to wait.

Best,

Francis

Philon

On 31 Jan 2020, at 17:24, Francis Augusto Medeiros-Logeay
 wrote:

Hi there,

I got successfully to replicate my mail server to another dovecot
install using dsync, mainly for redundancy, and it works great.

I want to try to install fts, as some of the mailboxes have tens of
thousands of messages, and it takes minutes to get some results when
searching via IMAP on a Roundcube interface.

I want to experiment with fts-solr first, and firstly on my
redundant server, ie., not on my main dovecot install. Is it ok to
do this? I ask because I am afraid of how this whole reindexing on
the redundant install will affect the production server.

Also, any tips on something else than fts-solr? I tried it once, but
it was so hard to get it right, so many configurations, java, etc.,
that I'd rather try something else. I also could try fts-elastic or
something like that, but, again, having to maintain an elasticsearch
install might use more resources than I think is worth. Any thoughts
on that?

Best,

-- Francis

Re: Email Server Backup Strategy

[Francis Augusto Medeiros-Logeay]

On 09.02.2020 19:08, Andrew McGlashan wrote:



If rsync is not fast... then how about doing a snapshot and then rsync
off the snapshot?  (I do rsync over lvm snapshots to back up file
systems cleanly) and, btw, perform file system dumps [ufsdump /
ufsrestore like, but dump and restore on Linux] of *non-mounted*
backup file systems fwiw.

I have other rsyncs that happen via rsnapshot during the day, I don't
worry about snapshotting them though, but I probably should


But thinking again about it, doesn't snapshotting every time leads to 
increased storage space?


Best,

Francis


0xEE41D33F.asc
Description: application/pgp-keys

Re: Email Server Backup Strategy

[Francis Augusto Medeiros-Logeay]

On 09.02.2020 19:08, Andrew McGlashan wrote:

Hi,

On 10/2/20 4:59 am, Francis Augusto Medeiros-Logeay wrote:

What I do:

1 - I ran a maildirlock command to my mail folder 2 - I then rsync
the folder where my maildirs are, as well as the indexes, to a
remote location


If rsync is not fast... then how about doing a snapshot and then rsync
off the snapshot?  (I do rsync over lvm snapshots to back up file
systems cleanly) and, btw, perform file system dumps [ufsdump /
ufsrestore like, but dump and restore on Linux] of *non-mounted*
backup file systems fwiw.


That's a good idea - In my case rsync is super fast, 1-2 min tops, so I 
can afford the lock for that time. But I surely should check more about 
snapshots.




I have other rsyncs that happen via rsnapshot during the day, I don't
worry about snapshotting them though, but I probably should


3 - I tar.gz the daily backup 4 - I kill the maildirlock process

I also use replication of my server so that I keep getting mail
when my main server isn't available.


You mean backup mx?


Yes. It is a work of beauty that thing :) I remember how cool it was to 
see mail being replicated when both servers are up and one responding 
for mail when one of the servers is down.


Best,

Francis

0xEE41D33F.asc
Description: application/pgp-keys

Re: Email Server Backup Strategy

[Francis Augusto Medeiros-Logeay]

What I do:

1 - I ran a maildirlock command to my mail folder
2 - I then rsync the folder where my maildirs are, as well as the 
indexes, to a remote location

3 - I tar.gz the daily backup
4 - I kill the maildirlock process

I also use replication of my server so that I keep getting mail when my 
main server isn't available.


As for postfix, I backup the database once a week, since it doesn't get 
changed that often.


Hope that helps!

Best,

Francis




On 09.02.2020 18:26, Esteban L wrote:

Hello,

Was thinking it would be wise to backup my server, in case anything
"bad" ever happened, so that I could quickly get up to speed.

1. clone the whole drive -- which might be the best to ensure I keep 
the

whole system working.

2. But, what about, if I just wanted to backup my Email server
components? Postfix - which I think is just config files, and Dovecot -
I think there is a doveadm backup for doing such things.

What do you guys do, recommend for backups??

Thanks

0xEE41D33F.asc
Description: application/pgp-keys


0xEE41D33F.asc
Description: application/pgp-keys

Re: Email Server Backup Strategy

[Francis Augusto Medeiros-Logeay]

What I do:

1 - I ran a maildirlock command to my mail folder
2 - I then rsync the folder where my maildirs are, as well as the 
indexes, to a remote location

3 - I tar.gz the daily backup
4 - I kill the maildirlock process

I also use replication of my server so that I keep getting mail when my 
main server isn't available.


As for postfix, I backup the database once a week, since it doesn't get 
changed that often.


Hope that helps!

Best,

Francis




On 09.02.2020 18:26, Esteban L wrote:

Hello,

Was thinking it would be wise to backup my server, in case anything
"bad" ever happened, so that I could quickly get up to speed.

1. clone the whole drive -- which might be the best to ensure I keep 
the

whole system working.

2. But, what about, if I just wanted to backup my Email server
components? Postfix - which I think is just config files, and Dovecot -
I think there is a doveadm backup for doing such things.

What do you guys do, recommend for backups??

Thanks

0xEE41D33F.asc
Description: application/pgp-keys

re: Dovecot - Upgrade Solr 7.7.2 to 8.4.1

[Francis Augusto Medeiros-Logeay]

On 1/23/20 7:03 AM, Domenico Pastore wrote:

So, with Dovecot is it possible to use Apache Solr 8.4?
High RAM usage is the only problem?



I'm using 8.4.0 and it works flawlessly.


I want to install fts-solr, but must tutorials are mentioning solr 
7.7.0. Any heads-up on what one must pay attention to when installing 
8.4.0? Do I need to update the version on the schemas, for example?


Best,

Francis

0xEE41D33F.asc
Description: application/pgp-keys

Re: Strategy for fts

[Francis Augusto Medeiros-Logeay]

---
Francis Augusto Medeiros-Logeay
Oslo, Norway

On 04.02.2020 22:55, Peter Chiochetti wrote:

Am 04.02.20 um 12:37 schrieb Peter Chiochetti:

Am 04.02.20 um 11:46 schrieb Francis Augusto Medeiros-Logeay:

Hi Philon,

Thanks a lot for your thoughts!

Can I ask you if using Solr improved things for you? I have a mailbox 
with 15 years of e-mail and searching things take a long time.


Here, SOLR itself searches a quarter million mails in split seconds 
and returns very good results. That is on a low memory average 
machine.




How much memory are you using, if I may ask? I have a really small 
server only with only 2GB. I am thinking about migrating it, but haven't 
done it so far, most likely to a 16GB instance.


Best,

Francis

0xEE41D33F.asc
Description: application/pgp-keys

Re: Strategy for fts

[Francis Augusto Medeiros-Logeay]

---
Francis Augusto Medeiros-Logeay
Oslo, Norway

On 04.02.2020 22:55, Peter Chiochetti wrote:

Am 04.02.20 um 12:37 schrieb Peter Chiochetti:

Am 04.02.20 um 11:46 schrieb Francis Augusto Medeiros-Logeay:

Hi Philon,

Thanks a lot for your thoughts!

Can I ask you if using Solr improved things for you? I have a mailbox 
with 15 years of e-mail and searching things take a long time.


Here, SOLR itself searches a quarter million mails in split seconds 
and returns very good results. That is on a low memory average 
machine.




How much memory are you using, if I may ask? I have a really small 
server only with only 2GB. I am thinking about migrating it, but haven't 
done it so far, most likely to a 16GB instance.


Best,

Francis

0xEE41D33F.asc
Description: application/pgp-keys


0xEE41D33F.asc
Description: application/pgp-keys

Re: Strategy for fts and Replication

[Francis Augusto Medeiros-Logeay]

Hi Philon,

Thanks a lot for your thoughts!

Can I ask you if using Solr improved things for you? I have a mailbox 
with 15 years of e-mail and searching things take a long time.


On 04.02.2020 09:39, Philon wrote:

Hi Francis,

next to fts-solr there was fts-lucene. But that Lucene there seems
heavily outdated why the Dovecot docs also suggest using Solr.
Elasticsearch probably is similar to Solr but the later is maintained
by Dovecot team.

I started with downloading the Solr binary distribution to Debian with
JRE preinstalled and things were running like after 10 min. Yes it’s a
bit more complicated to find the schema and edit things like header
size (in tips section). It’s running quite nicely since then and has
zero maintenance.


I will try again - I kept getting some weird errors, so I don't know if 
that's why I wasn't seing much of improvement.




As FTS indexes are separate in external Solr instance I’d guess that
it won’t interfere with dsync. What I don’t know is if dsync’ing would
trigger indexing. This brings me to wonder how one could actually
replicate the Solr instance!?


Good question. But what I thought about doing was to install FTS on my 
backup instance, and if things go fine, then I install an FTS instance 
on my production server - that is, if one doesn't interfere with the 
other.


I will give Solr another shot - my worries are mostly if Solr is 
supported on ARM (my prod instance is running on ARM) - I know 
Elasticsearch has an ARM build.


Ii thought about the Xapian engine, but since it requires dovecot 2.3, I 
will have to wait.


Best,

Francis




Philon

On 31 Jan 2020, at 17:24, Francis Augusto Medeiros-Logeay 
 wrote:


Hi there,

I got successfully to replicate my mail server to another dovecot 
install using dsync, mainly for redundancy, and it works great.


I want to try to install fts, as some of the mailboxes have tens of 
thousands of messages, and it takes minutes to get some results when 
searching via IMAP on a Roundcube interface.


I want to experiment with fts-solr first, and firstly on my redundant 
server, ie., not on my main dovecot install. Is it ok to do this? I 
ask because I am afraid of how this whole reindexing on the redundant 
install will affect the production server.


Also, any tips on something else than fts-solr? I tried it once, but 
it was so hard to get it right, so many configurations, java, etc., 
that I'd rather try something else. I also could try fts-elastic or 
something like that, but, again, having to maintain an elasticsearch 
install might use more resources than I think is worth. Any thoughts 
on that?


Best,

--
Francis


0xEE41D33F.asc
Description: application/pgp-keys

Strategy for fts and Replication

[Francis Augusto Medeiros-Logeay]

Hi there,

I got successfully to replicate my mail server to another dovecot 
install using dsync, mainly for redundancy, and it works great.


I want to try to install fts, as some of the mailboxes have tens of 
thousands of messages, and it takes minutes to get some results when 
searching via IMAP on a Roundcube interface.


I want to experiment with fts-solr first, and firstly on my redundant 
server, ie., not on my main dovecot install. Is it ok to do this? I ask 
because I am afraid of how this whole reindexing on the redundant 
install will affect the production server.


Also, any tips on something else than fts-solr? I tried it once, but it 
was so hard to get it right, so many configurations, java, etc., that 
I'd rather try something else. I also could try fts-elastic or something 
like that, but, again, having to maintain an elasticsearch install might 
use more resources than I think is worth. Any thoughts on that?


Best,

--
Francis