Re: how to configure imapsieve to be used per user

[Paul Kudla]

ok fair enuff

are you using a db to set the dir's

there is a master sieve (all) directory that handles the entire server 
(message duplicate supression etc)


mine is in

[17:26:12] mail18.scom.ca [root:0] /usr/local/etc/dovecot/sieve
# ll
total 38
drwxr-xr-x  2 vmail  vmail  uarch4B Apr  2  2022 .
drwxr-xr-x  5 root   wheel  uarch   29B Oct 27 07:41 ..
-rw-r--r--  1 vmail  vmail  uarch   97B Apr  2  2022 duplicates.sieve
-rw-r--r--  1 vmail  vmail  uarch  227B Apr  2  2022 duplicates.svbin

[17:26:17] mail18.scom.ca [root:0] /usr/local/etc/dovecot/sieve



from there each user (assuming dovecot config is correct will have it's 
own sieve folder under the maildir


example :

# mbox ab...@scom.ca

[17:27:24] mail18.scom.ca [root:0] /data/dovecot/users/scom.ca/ab...@scom.ca
# ll
total 293
drwx--   11 vmail  vmail  uarch   25B Oct 27 16:48 .
drwx--  164 vmail  vmail  uarch  164B Oct 27 06:52 ..
drwx--5 vmail  vmail  uarch8B Oct 27 16:48 .Drafts
drwx--5 vmail  vmail  uarch8B Oct 27 16:48 .Sent
drwx--5 vmail  vmail  uarch8B Oct 27 16:48 .Trash
-rw---1 vmail  vmail  uarch  1.4K Oct 26 16:49 .dovecot.lda-dupes
drwx--5 vmail  vmail  uarch7B Oct 27 16:48 
.dovecot.lda-dupes.locks

drwx--2 vmail  vmail  uarch2B Oct 26 16:48 cur
-rw---1 vmail  vmail  uarch  8.3K Oct 26 16:49 dovecot-uidlist
-rw---1 vmail  vmail  uarch8B Oct 26 16:49 dovecot-uidvalidity
-r--r--r--1 vmail  vmail  uarch0B Oct 26 16:48 
dovecot-uidvalidity.63599d11

-rw---1 vmail  vmail  uarch  3.7K Oct 27 16:48 dovecot.index
-rw---1 vmail  vmail  uarch   34K Oct 27 16:48 dovecot.index.cache
-rw---1 vmail  vmail  uarch  644B Oct 27 16:48 dovecot.index.log
-rw---1 vmail  vmail  uarch   40K Oct 27 16:48 dovecot.index.log.2
-rw---1 vmail  vmail  uarch  968B Oct 27 16:48 dovecot.list.index
-rw---1 vmail  vmail  uarch  1.7K Oct 27 16:48 
dovecot.list.index.log
-rw---1 vmail  vmail  uarch  8.2K Oct 27 16:48 
dovecot.list.index.log.2

-rw---1 vmail  vmail  uarch   96B Oct 26 16:48 dovecot.mailbox.log
drwx--2 vmail  vmail  uarch9B Oct 26 16:49 lucene-indexes
-rw---1 vmail  vmail  uarch0B Oct 26 16:48 maildirfolder
drwx--2 vmail  vmail  uarch  142B Oct 26 16:49 new
drwx--3 vmail  vmail  uarch6B Oct 26 16:49 sieve
-rw---1 vmail  vmail  uarch   29B Oct 26 16:48 subscriptions
drwx--2 vmail  vmail  uarch2B Oct 26 16:49 tmp


and then :

[17:27:42] mail18.scom.ca [root:0] 
/data/dovecot/users/scom.ca/ab...@scom.ca/sieve

# ll
total 67
drwx--   3 vmail  vmail  uarch6B Oct 26 16:49 .
drwx--  11 vmail  vmail  uarch   25B Oct 27 16:48 ..
lrwx--   1 vmail  vmail  uarch   13B Oct 27 16:48 .dovecot.sieve -> 
forward.sieve

-rw---   1 vmail  vmail  uarch  239B Oct 26 16:49 .dovecot.svbin
-rw---   1 vmail  vmail  uarch   31B Oct 26 16:48 forward.sieve
drwx--   2 vmail  vmail  uarch2B Oct 26 16:48 tmp

[17:27:44] mail18.scom.ca [root:0] 
/data/dovecot/users/scom.ca/ab...@scom.ca/sieve


for a user script to be active you need to set the script active (after 
uploading etc?) which creates a link from dovecot.sieve to the script 
and a .svbin file (i believe, this is an observation on my side)


if all this is setup properly then the script should execute?

please note my system is db driven and i am using virtual maildir's

if you are doing this manually then make sure the dovecot's user right's 
are correct


you are probably far enough along the set

mail_debug = yes

in dovecot.conf (remember to restart the server)

this should dump a wack of logging somewhere (file or syslog)

sieve or pigeonhole will be in there when you try to do something

fyi



fyi





Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 10/27/2022 4:06 PM, Sebastian Bachmann wrote:




On 27.10.2022 13:54, Paul Kudla wrote:
again may (probably not) what you are looking for but it at least 
gives another example(s)


No, actually I was looking for something different. The TO and me were 
looking for imapsieve examples and how they can be configured on a per 
user & per mailbox basis.


I tried now some things, and I'm at least one step further. The 
important parts seemed to be:


* Enable IMAP METADATA
* Set `imapsieve_url = sieve://server:4190` (is that correct?)

Now you can add the metadata, for example to the mailbox "test":
a SETMETADATA test (/shared/imapsieve/script "sieve/imap.sieve")

However, from this point on it does not work. I created a very simple 
example, which should simple copy any mail that is moved into the folder 
(right?):


require ["copy"];
redirect :copy "some_other_email_address";

but it looks like the script is never 

Re: how to configure imapsieve to be used per user

[Sebastian Bachmann]

On 27.10.2022 13:54, Paul Kudla wrote:
again may (probably not) what you are looking for but it at least gives 
another example(s)


No, actually I was looking for something different. The TO and me were 
looking for imapsieve examples and how they can be configured on a per 
user & per mailbox basis.


I tried now some things, and I'm at least one step further. The 
important parts seemed to be:


* Enable IMAP METADATA
* Set `imapsieve_url = sieve://server:4190` (is that correct?)

Now you can add the metadata, for example to the mailbox "test":
a SETMETADATA test (/shared/imapsieve/script "sieve/imap.sieve")

However, from this point on it does not work. I created a very simple 
example, which should simple copy any mail that is moved into the folder 
(right?):


require ["copy"];
redirect :copy "some_other_email_address";

but it looks like the script is never started.
Is this because I mis-configured the path? Or something else? It is in 
the same folder as I store the other scripts, i.e., ~/sieve (the normal 
sieve scripts work fine)


-Sebastian

oauth2 + postgres = Panic: file http-client.c: line 646 (http_client_context_close)

[Sebastiano Degan]

I could to reply to the original thread:
https://www.mail-archive.com/dovecot@dovecot.org/msg85204.html

But I managed to get a stack trace of the error:

root@mail:~ # lldb /usr/local/libexec/dovecot/auth
(lldb) target create "/usr/local/libexec/dovecot/auth"
Current executable set to '/usr/local/libexec/dovecot/auth' (x86_64).
(lldb) run
Process 54354 launched: '/usr/local/libexec/dovecot/auth' (x86_64)
Error: stats: open(old-stats-user) failed: No such file or directory
Panic: file http-client.c: line 646 (http_client_context_close): assertion
failed: (cctx->clients_list == NULL)
Process 54354 stopped
* thread #1, name = 'auth', stop reason = signal SIGABRT
frame #0: 0x00080159f33a libc.so.7`__sys_thr_kill + 10
libc.so.7`__sys_thr_kill:
->  0x80159f33a <+10>: jb 0x80159d498
0x80159f340 <+16>: retq
0x80159f341:   int3
0x80159f342:   int3
(lldb) bt all
* thread #1, name = 'auth', stop reason = signal SIGABRT
  * frame #0: 0x00080159f33a libc.so.7`__sys_thr_kill + 10
frame #1: 0x000801517c74 libc.so.7`__raise + 52
frame #2: 0x0008015c9109 libc.so.7`abort + 73
frame #3: 0x0008012e606b
libdovecot.so.0`default_fatal_finish(type=LOG_TYPE_PANIC, status=0) at
failures.c:465:3
frame #4: 0x0008012e3f7c
libdovecot.so.0`fatal_handler_real(ctx=0x7fffe610, format="file %s:
line %d (%s): assertion failed: (%s)", args=0x7fffe5f0) at
failures.c:477:2
frame #5: 0x0008012e3f14
libdovecot.so.0`default_fatal_handler(ctx=0x7fffe610, format="file
%s: line %d (%s): assertion failed: (%s)", args=0x7fffe5f0) at
failures.c:485:2
frame #6: 0x0008012e4253 libdovecot.so.0`i_panic(format="file %s:
line %d (%s): assertion failed: (%s)") at failures.c:530:2
frame #7: 0x00080124f2ed
libdovecot.so.0`http_client_context_close(cctx=0x00080205c048) at
http-client.c:646:2
frame #8: 0x00080124ef22
libdovecot.so.0`http_client_global_context_ioloop_switched(prev_ioloop=0x000802030640)
at http-client.c:710:3
frame #9: 0x0008013102e4
libdovecot.so.0`io_loop_set_current(ioloop=0x) at
ioloop.c:982:4
frame #10: 0x010aa0be
auth`driver_pgsql_set_state(db=0x000802092200, state=SQL_DB_STATE_IDLE)
at driver-pgsql.c:103:3
frame #11: 0x010a9ff0
auth`connect_callback(db=0x000802092200) at driver-pgsql.c:206:3
frame #12: 0x00080130f3b2
libdovecot.so.0`io_loop_call_io(io=0x0008020105c0) at ioloop.c:737:2
frame #13: 0x000801313e0b
libdovecot.so.0`io_loop_handler_run_internal(ioloop=0x000802030640) at
ioloop-kqueue.c:164:4
frame #14: 0x00080130fa15
libdovecot.so.0`io_loop_handler_run(ioloop=0x000802030640) at
ioloop.c:789:2
frame #15: 0x00080130f8fa
libdovecot.so.0`io_loop_run(ioloop=0x000802030640) at ioloop.c:762:3
frame #16: 0x010a7eef
auth`driver_pgsql_wait(_db=0x000802092200) at driver-pgsql.c:1279:3
frame #17: 0x010a4c59 auth`sql_wait(db=0x000802092200) at
sql-api.c:820:3
frame #18: 0x010a7fc0
auth`driver_pgsql_get_flags(db=0x000802092200) at driver-pgsql.c:306:3
frame #19: 0x010a2fa9 auth`sql_get_flags(db=0x000802092200)
at sql-api.c:171:10
frame #20: 0x010a585e
auth`driver_sqlpool_get_flags(_db=0x000802092000) at
driver-sqlpool.c:450:9
frame #21: 0x010a2fa9 auth`sql_get_flags(db=0x000802092000)
at sql-api.c:171:10
frame #22: 0x0108c675
auth`passdb_sql_init(_module=0x000802041ca8) at passdb-sql.c:281:10
frame #23: 0x0107d577
auth`passdb_init(passdb=0x000802041ca8) at passdb.c:247:3
frame #24: 0x01052a89
auth`auth_passdb_init(passdb=0x000802041ba8) at auth.c:308:2
frame #25: 0x01051dbc auth`auth_init(auth=0x000802041b70)
at auth.c:322:3
frame #26: 0x01051d21 auth`auths_init at auth.c:427:3
frame #27: 0x01050610 auth`main_init at main.c:242:2
frame #28: 0x010503ad auth`main(argc=1,
argv=0x7fffeaf8) at main.c:391:2
frame #29: 0x0104ff7d auth`_start(ap=,
cleanup=) at crt1_c.c:75:7

Re: Mailing list IMAP archive.

[Narcis Garcia]

El 27/10/22 a les 18:04, dove...@ptld.com ha escrit:
I am trying to use the IMAP mailing list archive described on 
https://www.dovecot.org/mailing-lists/
Is this feature still functioning? Using Thunderbird i get an error 
message that the connection was refused.


Try using dovecot.org:imaps (port 993)


Thanks, that worked.

For others using Thunderbird, i set:
     Server: dovecot.org
     User: anonymous
     Port: 993
     Connection: SSL/TLS
     Auth: Normal password



Oh nice!!
But 10+3+500 messages!


--

Narcis Garcia

__
I'm using this dedicated address because personal addresses aren't 
masked enough at this mail public archive. Public archive administrator 
should fix this against automated addresses collectors.

Re: Mailing list IMAP archive.

[dovecot]

I am trying to use the IMAP mailing list archive described on 
https://www.dovecot.org/mailing-lists/
Is this feature still functioning? Using Thunderbird i get an error message 
that the connection was refused.


Try using dovecot.org:imaps (port 993)


Thanks, that worked.

For others using Thunderbird, i set:
Server: dovecot.org
User: anonymous
Port: 993
Connection: SSL/TLS
Auth: Normal password

Re: Mailing list IMAP archive.

[Aki Tuomi]

> On 27/10/2022 18:36 EEST dove...@ptld.com wrote:
> 
>  
> I am trying to use the IMAP mailing list archive described on 
> https://www.dovecot.org/mailing-lists/
> 
> Is this feature still functioning? Using Thunderbird i get an error message 
> that the connection was refused.
> I don't know if Thunderbird isn't functioning properly (with handling no 
> password) or if the service is down.
> 
> My end goal is i want to search the archive for solutions before asking the 
> list.
> Is there a browser based option to search the archives?
> I didn't see a search option on https://dovecot.org/pipermail/dovecot/

Try using dovecot.org:imaps (port 993)

Aki

Mailing list IMAP archive.

[dovecot]

I am trying to use the IMAP mailing list archive described on 
https://www.dovecot.org/mailing-lists/

Is this feature still functioning? Using Thunderbird i get an error message 
that the connection was refused.
I don't know if Thunderbird isn't functioning properly (with handling no 
password) or if the service is down.

My end goal is i want to search the archive for solutions before asking the 
list.
Is there a browser based option to search the archives?
I didn't see a search option on https://dovecot.org/pipermail/dovecot/

Re: how to configure imapsieve to be used per user

[Paul Kudla]

My apologies to the response eariler

I was making the assumption that you were using pigeonholes

it needs to be compiled seperately after making dovecot's server installs

basically the pigeonholes has to be compiled against the dovecot version 
you are running


after which my post info will be valid.

fyi .





Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 10/27/2022 9:48 AM, dovecot-boun...@dovecot.org wrote:


On 2022-10-27 02:28, Stephan Bosch wrote:

On 24-10-2022 12:00, Sebastian Bachmann wrote:
according to the documentation, this has to be added to the IMAP 
METADATA dict per mailbox 
(https://doc.dovecot.org/configuration_manual/imap_metadata/):


https://doc.dovecot.org/configuration_manual/sieve/plugins/imapsieve/ 
says:
The basic IMAPSIEVE capability allows attaching a Sieve script to a 
mailbox for any mailbox by setting a special IMAP METADATA entry. 
This way, users can configure Sieve scripts that are run for IMAP 
events in their mailboxes.
But I can not find any example how this should work, neither which 
client supports setting those things.
My guess is that these keys are used: 
https://www.iana.org/assignments/imap-metadata/imap-metadata.xhtml#imap-metadata-2


I would also be interested to know if and how that works, especially 
if you can add a rule when moving mails (from anywhere) to a certain 
mailbox for a single user.


The basic capability works according to the specification: 
https://www.rfc-editor.org/rfc/rfc6785

This allows the users to configure these scripts.

If you want to arrange this solely at the administrator's discretion, 
you can use the _before/_after settings documented in 
https://doc.dovecot.org/configuration_manual/sieve/plugins/imapsieve




Best,
Sebastian

On 17.10.2022 12:46, Marc wrote:


I only see configurations that are active for all users, how to 
configure this in the user sieve rules. I only need this for 
specific users.




Why dont you use pigeonholes?

Also, I recommend to look for Symlink creation titled post here in the 
mailing list, there is few points in about setting up per user sieve 
scripts that will be helpful to you.


Also, there is other posts on how to setup sieve for per user scripts.

Zakaria.

Re: how to configure imapsieve to be used per user

[hi]

On 2022-10-27 02:28, Stephan Bosch wrote:

On 24-10-2022 12:00, Sebastian Bachmann wrote:
according to the documentation, this has to be added to the IMAP 
METADATA dict per mailbox 
(https://doc.dovecot.org/configuration_manual/imap_metadata/):


https://doc.dovecot.org/configuration_manual/sieve/plugins/imapsieve/ 
says:
The basic IMAPSIEVE capability allows attaching a Sieve script to a 
mailbox for any mailbox by setting a special IMAP METADATA entry. 
This way, users can configure Sieve scripts that are run for IMAP 
events in their mailboxes.
But I can not find any example how this should work, neither which 
client supports setting those things.
My guess is that these keys are used: 
https://www.iana.org/assignments/imap-metadata/imap-metadata.xhtml#imap-metadata-2


I would also be interested to know if and how that works, especially 
if you can add a rule when moving mails (from anywhere) to a certain 
mailbox for a single user.


The basic capability works according to the specification: 
https://www.rfc-editor.org/rfc/rfc6785

This allows the users to configure these scripts.

If you want to arrange this solely at the administrator's discretion, 
you can use the _before/_after settings documented in 
https://doc.dovecot.org/configuration_manual/sieve/plugins/imapsieve




Best,
Sebastian

On 17.10.2022 12:46, Marc wrote:


I only see configurations that are active for all users, how to 
configure this in the user sieve rules. I only need this for specific 
users.




Why dont you use pigeonholes?

Also, I recommend to look for Symlink creation titled post here in the 
mailing list, there is few points in about setting up per user sieve 
scripts that will be helpful to you.


Also, there is other posts on how to setup sieve for per user scripts.

Zakaria.

Re: how to configure imapsieve to be used per user

[Paul Kudla]

ok a few things about sieve

although it is a pain it is usually (on a per user basis) better to 
access the sieve scripting through thunderbird's plugin sieve or 
something similiar as it will sort out checksums, syntax etc.


see :

https://github.com/thsmi/sieve/

https://www.pair.com/support/kb/sieve-syntax-and-common-recipes/

https://wiki.dovecot.org/Pigeonhole/ManageSieve/Troubleshooting

also RFC

https://datatracker.ietf.org/doc/html/rfc5804#page-3



Next and all though way more complicated since you need to calculate 
checksums etc i use a telnet python script that generates forward's etc


again may (probably not) what you are looking for but it at least gives 
another example(s)


python2 code below:

last but not least there is an RFC avaliable for sieve scripting but 
again its at a programming level and may not be overally useful ? (note 
link above)


don't feel bad it took me a while to figure this out as well.

for any of this to work you need the sieve listener port running on the 
mail server


make sure you can telnet to mailserver:2000 (older) or port 4190 
(current) depending how you are configured for sieve before going any 
further.


___

dovecot.conf

protocols = imap pop3 lmtp sieve

protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = moni...@scom.ca
}


  sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
  #sieve = ~/.dovecot.sieve
  sieve_duplicate_default_period = 1h
  sieve_duplicate_max_period = 1h
  sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
  sieve_global_dir = /usr/local/etc/dovecot/sieve
  sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve


service managesieve-login {
  process_limit = 1000
  vsz_limit = 1g
  inet_listener sieve {
port = 4190
  }
}

protocol sieve {
  managesieve_implementation_string = Dovecot Pigeonhole
  managesieve_max_line_length = 65536
}


___

above is my sieve conf alter accordingly if needed 



below are examples of my sieve processing scripts for vacation notices 
and forward's, this data comes from my django project model but should 
be pretty clear what the code is doing.


note : you need to base64 the auth username & password to login to the 
users sieve account.


note : \r\n needs to be used as a line terminator and calculated 
accordingly. See count & count2 below. Basically \r\n = 1


you can expand on this 



if self.vacation_active == True and dontupdate != True:
#debug = [debug,server,port,count,label,pid]
debug = 'syslog,10.228.0.6,514,0,sieve,0'
			log_debug (debug, 'Sieve (Vacation) : Do Not Update Status for : %s' 
%dontupdate )

import base64,telnetlib
log_debug (debug, 'Enabling Sieve for : %s' 
%self.username )
auth = '\0%s\0%s' %(self.username,self.password)
log_debug (debug, 'Auth : %s' %auth)
auth = base64.b64encode(auth)
log_debug (debug, 'Auth Encoded : %s' %auth)
from telnetlib import Telnet
tn =  Telnet('10.220.0.18', 4190)
connect = tn.read_until('OK',5)
log_debug (debug, 'Connect : \n%s\n' %connect)
authout = 'AUTHENTICATE "PLAIN" "%s"\n'%auth
log_debug (debug, 'Authout : %s' %authout)
tn.write(authout)
status = tn.expect(['OK','NO'],5)
log_debug (debug, 'Auth : %s' %str(status) )
tn.write('LISTSCRIPTS\r\n')
status = tn.expect(['OK','NO'],5)
log_debug (debug, 'Current Scripts : %s' %str(status) )
#Send a Script
script = 'keep;\r\nredirect 
"vacationprocess...@scom.ca";\r\n'
count = len(script)
count2 = script.count('\r\n')
log_debug (debug, 'Count : %s' %count)
log_debug (debug, 'Count 2 : %s'  %count2)
init = 'PUTSCRIPT "forward" {%s+}\r\n' %(count - count2)
log_debug (debug, 'Init : %s' %init)
tn.write ( init )
log_debug (debug, 'Script Len : %s' %len(script) )
log_debug (debug, 'Script : %s' %script)
tn.write( script )
status = tn.expect(['OK','NO'],5)
log_debug (debug, 'Write Status : %s' %str(status) )
log_debug (debug,  'Setting Active' )
tn.write('SETACTIVE 

Re: The end of Dovecot Director?

[Jan Bramkamp]

On 27.10.22 04:24, Timo Sirainen wrote:

Director never worked especially well, and for most use cases it's just 
unnecessarily complex. I think usually it could be replaced with:

  * Database (sql/ldap/whatever) containing user -> backend table.
  * Configure Dovecot proxy to use this database as passdb.
  * For HA change dovemon to update the database if backend is down to move 
users elsewhere
  * When backend comes up, move users into it. Set delay_until extra field for 
user in passdb to 5 seconds into future and kick the user in its old backend 
(e.g. via doveadm HTTP API).

All this can be done with existing Dovecot. Should be much easier to build a 
project doing this than forking director.
Thank you for putting what is about to be lost to the community edition 
into an operational perspectiv: no reason to panic. Nobody is taking 
replicated active-passive pairs from small to medium scale operators. 
Neither are the hooks required for more fancy load balancing and 
steering on the chopping block.

Re: The end of Dovecot Director?

[hi]

On 2022-10-27 08:31, William Edwards wrote:


Op 27 okt. 2022 om 04:25 heeft Timo Sirainen  het 
volgende geschreven:


Director never worked especially well, and for most use cases it's 
just unnecessarily complex. I think usually it could be replaced with:


* Database (sql/ldap/whatever) containing user -> backend table.
* Configure Dovecot proxy to use this database as passdb.
* For HA change dovemon to update the database if backend is down to 
move users elsewhere
* When backend comes up, move users into it. Set delay_until extra 
field for user in passdb to 5 seconds into future and kick the user in 
its old backend (e.g. via doveadm HTTP API).


All this can be done with existing Dovecot. Should be much easier to 
build a project doing this than forking director.


This is my train of thought as well. I believe the following would 
suffice for most setups.


A database with:

- Current vhost count per backend server. Alternatively, count the 
temporary user mappings.

- Backend servers.
- Temporary user mappings between user - backend server.

This database is accessible by all Dovecot proxies in case there’s 
multiple.


Steps when receiving a login:

- Check if a temporary user mapping exists.
- If so, proxy to the backend server in the temporary mapping. (To do: 
clean up mappings.)
- If not, pick the backend server with the lowest vhost count, create a 
temporary mapping, then increase the vhost count of the chosen backend 
server.


A monitoring service up/downs backend servers. E.g. by checking the 
port that we proxy to for each backend server. When a backend server is 
set to down, kick the user to force a reconnection. (Is that how 
Director ‘moves’ users?)


Here is my alternative input as well using database cluster/file.

Create connection mappings table in database cluster where each row must 
be containing user id, backend id and frontend id and agent hash, 
alternatively mappings file containing such info and synced across all 
servers.


Incorporate multiple simultaneous mappings using agent hash which can be 
useful e.g. in the event of using client apps from several devices, in 
the IMAP proxy perhaps update the first row agent hash which doesnt have 
hash and matching frontend and user id in post login requests.


Create service in each backend, monitoring login and logout entries, and 
whenever there is one, add the relevant user and frontend row in 
mappings table/file. In the event of remove just mark one matching entry 
with exclusion to unknown agent hash as soft removed.


In load balancing solution, for SMTP/IMAP connections, use perhaps a lua 
script, to check mappings in database or file, and find which backend 
user was logged to, and alongside generate user agent hash perhaps using 
base64 encoding to locate exact client connection backend row in 
mappings where several entries might be present, and proxy the incoming 
request to it, uncheck soft removed if same backend using same user 
agent hash, if there is no mappings, use the normal load balancing 
method which in post login requests its mappings will be automatically 
created.


Zakaria.

Re: Catch all for Metadata storage in SQL database

[Aki Tuomi]

> On 03/05/2021 09:10 EEST Aki Tuomi  wrote:
> 
>  
> > On 30/04/2021 09:38 Steffen Kaiser  wrote:
> > 
> >  
> > Hi,
> > 
> > the
> > 
> > https://doc.dovecot.org/configuration_manual/imap_metadata/
> > 
> > sample uses
> > 
> > mail_attribute_dict = file:%h/Maildir/dovecot-attributes
> > 
> > which stores all keys=value pairs in the file.
> > 
> > http://dovecot.2317879.n4.nabble.com/Dovecot-v2-3-9-3-HTTP-API-Endpoint-for-mailbox-cryptokey-operations-td70801.html
> > 
> > uses a SQL dict, however very specific ones. How can I use a SQL dict to 
> > store _all_ keys, as with a file based storage?
> > 
> > I cannot find a documentation for "pattern" specification, that works as 
> > "catch all", in order to store anything not catched by patterns into the 
> > database.
> > 
> > https://wiki.dovecot.org/Dictionary does not give any hint (in my eyes).
> > 
> > 
> > 
> > -- 
> > Steffen Kaiser
> 
> 
> for one, you need to use proxy::metadata
> 
> then define
> 
> dict {
>metadata = mysql:/path/to/config
> }
> 
> then you need the mapping file, which you could use something like:
> 
> connect = host=localhost dbname=dovecot user=dovecot password=dovecot
> map {
>  pattern = priv/$key
>  fields {
>meta_key = $key
>  }
>  table = meta
>  username_field = username
>  value_field = value
> }
> 
> with
> 
> CREATE TABLE meta (
>username VARCHAR(255) NOT NULL,
>meta_key VARCHAR(255) NOT NULL,
>value VARCHAR(255),
>PRIMARY KEY(username, `key`)
> );
> 
> Hope this helps.
> 
> Aki

Hi!

(sorry for prev email, MUA acted up somehow)

There is now better documentation on how to achieve this: 
https://doc.dovecot.org/configuration_manual/dict/#sql-dict-with-mail-attribute-dict

Aki

Re: Catch all for Metadata storage in SQL database

[Aki Tuomi]

> On 03/05/2021 09:10 EEST Aki Tuomi  wrote:
> 
>  
> > On 30/04/2021 09:38 Steffen Kaiser  wrote:
> > 
> >  
> > Hi,
> > 
> > the
> > 
> > https://doc.dovecot.org/configuration_manual/imap_metadata/
> > 
> > sample uses
> > 
> > mail_attribute_dict = file:%h/Maildir/dovecot-attributes
> > 
> > which stores all keys=value pairs in the file.
> > 
> > http://dovecot.2317879.n4.nabble.com/Dovecot-v2-3-9-3-HTTP-API-Endpoint-for-mailbox-cryptokey-operations-td70801.html
> > 
> > uses a SQL dict, however very specific ones. How can I use a SQL dict to 
> > store _all_ keys, as with a file based storage?
> > 
> > I cannot find a documentation for "pattern" specification, that works as 
> > "catch all", in order to store anything not catched by patterns into the 
> > database.
> > 
> > https://wiki.dovecot.org/Dictionary does not give any hint (in my eyes).
> > 
> > 
> > 
> > -- 
> > Steffen Kaiser
> 
> 
> for one, you need to use proxy::metadata
> 
> then define
> 
> dict {
>metadata = mysql:/path/to/config
> }
> 
> then you need the mapping file, which you could use something like:
> 
> connect = host=localhost dbname=dovecot user=dovecot password=dovecot
> map {
>  pattern = priv/$key
>  fields {
>meta_key = $key
>  }
>  table = meta
>  username_field = username
>  value_field = value
> }
> 
> with
> 
> CREATE TABLE meta (
>username VARCHAR(255) NOT NULL,
>meta_key VARCHAR(255) NOT NULL,
>value VARCHAR(255),
>PRIMARY KEY(username, `key`)
> );
> 
> Hope this helps.
> 
> Aki


Hi!

There is now better documentation on how to achieve this: 
https://doc.dovecot.org/configuration_manual/dict/#sql-dict-with-mail-attribute-dict

Aki

Re: The end of Dovecot Director?

[hi]

On 2022-10-27 08:31, William Edwards wrote:


Op 27 okt. 2022 om 04:25 heeft Timo Sirainen  het 
volgende geschreven:


Director never worked especially well, and for most use cases it's 
just unnecessarily complex. I think usually it could be replaced with:


* Database (sql/ldap/whatever) containing user -> backend table.
* Configure Dovecot proxy to use this database as passdb.
* For HA change dovemon to update the database if backend is down to 
move users elsewhere
* When backend comes up, move users into it. Set delay_until extra 
field for user in passdb to 5 seconds into future and kick the user in 
its old backend (e.g. via doveadm HTTP API).


All this can be done with existing Dovecot. Should be much easier to 
build a project doing this than forking director.


This is my train of thought as well. I believe the following would 
suffice for most setups.


A database with:

- Current vhost count per backend server. Alternatively, count the 
temporary user mappings.

- Backend servers.
- Temporary user mappings between user - backend server.

This database is accessible by all Dovecot proxies in case there’s 
multiple.


Steps when receiving a login:

- Check if a temporary user mapping exists.
- If so, proxy to the backend server in the temporary mapping. (To do: 
clean up mappings.)
- If not, pick the backend server with the lowest vhost count, create a 
temporary mapping, then increase the vhost count of the chosen backend 
server.


A monitoring service up/downs backend servers. E.g. by checking the 
port that we proxy to for each backend server. When a backend server is 
set to down, kick the user to force a reconnection. (Is that how 
Director ‘moves’ users?)


Here is my alternative input as well using database cluster/file.

Create connection mappings table in database cluster where each row must 
be containing user id, backend id and frontend id and agent hash, 
alternatively mappings file containing such info and synced across all 
servers.


Incorporate multiple simultaneous mappings using agent hash which can be 
useful e.g. in the event of using client apps from several devices, in 
the IMAP proxy perhaps update the first row agent hash which doesnt have 
hash and matching frontend and user id in post login requests.


Create service in each backend, monitoring login and logout entries, and 
whenever there is one, add the relevant user and frontend row in 
mappings table/file. In the event of remove just mark one matching entry 
with exclusion to unknown agent hash as soft removed.


In load balancing solution, for SMTP/IMAP connections, use perhaps a lua 
script, to check mappings in database or file, and find which backend 
user was logged to, and alongside generate user agent hash perhaps using 
base64 encoding to locate exact client connection backend row in 
mappings where several entries might be present, and proxy the incoming 
request to it, uncheck soft removed if same backend using same user 
agent hash, if there is no mappings, use the normal load balancing 
method which in post login requests its mappings will be automatically 
created.


Zakaria.

Re: The end of Dovecot Director?

[William Edwards]


> Op 27 okt. 2022 om 04:25 heeft Timo Sirainen  het volgende 
> geschreven:
> 
> Director never worked especially well, and for most use cases it's just 
> unnecessarily complex. I think usually it could be replaced with:
> 
> * Database (sql/ldap/whatever) containing user -> backend table.
> * Configure Dovecot proxy to use this database as passdb.
> * For HA change dovemon to update the database if backend is down to move 
> users elsewhere
> * When backend comes up, move users into it. Set delay_until extra field for 
> user in passdb to 5 seconds into future and kick the user in its old backend 
> (e.g. via doveadm HTTP API).
> 
> All this can be done with existing Dovecot. Should be much easier to build a 
> project doing this than forking director.

This is my train of thought as well. I believe the following would suffice for 
most setups.

A database with:

- Current vhost count per backend server. Alternatively, count the temporary 
user mappings.
- Backend servers.
- Temporary user mappings between user - backend server.

This database is accessible by all Dovecot proxies in case there’s multiple.

Steps when receiving a login:

- Check if a temporary user mapping exists.
- If so, proxy to the backend server in the temporary mapping. (To do: clean up 
mappings.)
- If not, pick the backend server with the lowest vhost count, create a 
temporary mapping, then increase the vhost count of the chosen backend server.

A monitoring service up/downs backend servers. E.g. by checking the port that 
we proxy to for each backend server. When a backend server is set to down, kick 
the user to force a reconnection. (Is that how Director ‘moves’ users?)