Re: Spam learning for rspamd

2020-10-12 Thread Gedalya 
On 10/13/20 8:49 AM, Dan Egli wrote:
>
> I'm quite new to Dovecot, so forgive me if this is a simple question. I've 
> got rspamd running, and it's rewriting the subject of many messages as spam 
> even when they are not. I've moved things out of the spam folder, which I was 
> under the impression would teach rspamd since I've connected a sieve script 
> that is supposed to call rspamd's learning tool, but nothing is happening. 
> I'm really at a loss as to where to even begin searching for an answer, so 
> any help is appreciated!
>
> -- 
> Dan Egli
> On my Test server

At first we'd want to see your current configuration, sieve scripts etc.

Re: Spam learning for rspamd

2020-10-12 Thread Aki Tuomi 


> On 13/10/2020 07:33 Dan Egli  wrote:
> 
> 
> On 10/12/2020 7:28 PM, Gedalya wrote:
> 
> > On 10/13/20 8:49 AM, Dan Egli wrote:
> > 
> > > I'm quite new to Dovecot, so forgive me if this is a simple question. 
> > > I've got rspamd running, and it's rewriting the subject of many messages 
> > > as spam even when they are not. I've moved things out of the spam folder, 
> > > which I was under the impression would teach rspamd since I've connected 
> > > a sieve script that is supposed to call rspamd's learning tool, but 
> > > nothing is happening. I'm really at a loss as to where to even begin 
> > > searching for an answer, so any help is appreciated!
> > > -- 
> > > Dan Egli
> > > On my Test server
> > At first we'd want to see your current configuration, sieve scripts etc.
> > 
> > 
> Find them attached to this message. The contents of /etc/dovecot and 
> /usr/lib/dovecot/sieve, packaged in a tarball. Hope that helps. And how do 
> you train rspamd when it doesn't put the files in the spam folder but still 
> gives them a spam warning in the subject? (i.e. [*SP4M*])??
> 
> -- 
> Dan Egli
> On my Test server


Try this guide:

https://doc.dovecot.org/configuration_manual/howto/antispam_with_sieve/

Aki

Re: Spam learning for rspamd

2020-10-12 Thread Dan Egli 
On 10/12/2020 7:28 PM, Gedalya wrote:
> On 10/13/20 8:49 AM, Dan Egli wrote:
>>
>> I'm quite new to Dovecot, so forgive me if this is a simple question.
>> I've got rspamd running, and it's rewriting the subject of many
>> messages as spam even when they are not. I've moved things out of the
>> spam folder, which I was under the impression would teach rspamd
>> since I've connected a sieve script that is supposed to call rspamd's
>> learning tool, but nothing is happening. I'm really at a loss as to
>> where to even begin searching for an answer, so any help is appreciated!
>>
>> -- 
>> Dan Egli
>> On my Test server
>
> At first we'd want to see your current configuration, sieve scripts etc.
>
>
Find them attached to this message. The contents of /etc/dovecot and
/usr/lib/dovecot/sieve, packaged in a tarball. Hope that helps. And how
do you train rspamd when it doesn't put the files in the spam folder but
still gives them a spam warning in the subject? (i.e. [*SP4M*])??

-- 
Dan Egli
On my Test server



dovecot.tbz2
Description: Binary data


OpenPGP_0xF8A7B3F2AAB08F9D.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature

Re: DKIM fail if WHM adds Message-ID, should be Message-Id

2020-10-12 Thread Adi Pircalabu 

On 12-10-2020 19:16, Robert Martin wrote:

I created a client library to send emails for a webapp.

After connecting to the SMTP server with credential setup in CPANEL,
and then do NOT add Message-Id header, the DKIM signature 'h' record
created by dovecot/WHM is wrong, and a Message-ID (with a capital D)
header is added, invalidating the generated DKIM signature value.

This causes outlook, yahoo, gmail and other email recipients to add
'dkim:fail' to the message, and thus relegate it to junk or spam.

The work around is to add to the message a Message-Id with a  little
'd' header.  Then the SMTP server processes the email with the correct
generated DKIM, correct DKIM 'h' record and does not add a Message-ID
header.

My SMTP hosting providers that run the WHM/dovecot/CPANEL software are
refusing to raise this as a bug and have requested that I do it.


Hardly a bug imo. Best to add the Message-Id header from your library 
and this will become a non-issue. There are certain filters who don't 
quite like emails without message-id header, or incorrectly formatting 
ones because this is an usual fingerprint of a broken client / spam bot, 
this is why your provider is adding that header.


--
Adi Pircalabu

Move older messages to Old folder after a bunch of changes (sql + other stuff)

2020-10-12 Thread Chris Bennett 
I'm starting to get things better done now.
Moved to all virtual users over IMAP.
Using a SQL database for users/passwords.
Setting up folders working properly with neomutt.

Right now, that leaves me with just new mail in the INBOX, and other
folders.
I have a bunch of old mail in each users Maildirs in their real home
folders.
Right now, I'd like to move all the old mail into a folder (doesn't
really matter what it's called).

Right now is actually a perfect time to do this by good luck in timing.

How should I go about creating a new folder for those messages and then
adding the old mail from system user home Maildir folders to the virtual
users folders I'm using?

Thanks for any help,
Chris Bennett

Spam learning for rspamd

2020-10-12 Thread Dan Egli 
I'm quite new to Dovecot, so forgive me if this is a simple question.
I've got rspamd running, and it's rewriting the subject of many messages
as spam even when they are not. I've moved things out of the spam
folder, which I was under the impression would teach rspamd since I've
connected a sieve script that is supposed to call rspamd's learning
tool, but nothing is happening. I'm really at a loss as to where to even
begin searching for an answer, so any help is appreciated!

-- 
Dan Egli
On my Test server



OpenPGP_0xF8A7B3F2AAB08F9D.asc
Description: application/pgp-keys


OpenPGP_signature
Description: OpenPGP digital signature

RE: Preparing for replication: dsync-local(testaccount): Panic: file mbox-lock.c

2020-10-12 Thread Marc Roos 


Maybe a bug? 
I deleted the whole tree on the destination server, ran 
doveadm sync -D -1 -n inbox -u testacc tcp:mailxx.local:542
Only the directory structure of mdbox was created, no files.

2nd time I run:
doveadm sync -D -1 -n inbox -u testacc tcp:mailxx.local:542

The files are created in storage. 

What makes the m.XXX reset? I thought this would start at m.001 
after deleting the whole home dir


[1]
[@storage]# ls -cs1t
3928 m.737
4032 m.734
5868 m.735
5868 m.736
4100 m.733
4100 m.717
4100 m.718
4100 m.719
4100 m.720
4100 m.721
4100 m.722
4100 m.723
4100 m.724
4100 m.725
4100 m.726
4100 m.727
4100 m.728
4100 m.729
4100 m.730
4100 m.731
4100 m.732
4052 m.716

[2]
[@]# ls -cs1t mdbox/storage/
total 93448
3928 m.759
4032 m.756
4100 m.741
4100 m.742
4100 m.743
4100 m.744
4100 m.745
4100 m.746
4100 m.747
4100 m.748
4100 m.749
4100 m.750
4100 m.751
4100 m.752
4100 m.753
4100 m.754
4100 m.755
5868 m.757
5868 m.758
4100 m.739
4100 m.740
4052 m.738


-Original Message-
Subject: Re: Preparing for replication: dsync-local(testaccount): Panic: 
file mbox-lock.c

Replication is not supported with mbox format. You can only do 
unidirectional sync out of mbox. 

Aki 


I am preparing a bit for setting up replication. However when I 
manually 
try to dsync an account, the first time I execute this[1] command 
it 
seems to be ok. The 2nd time I am getting this error[2]. 
If I add -1 (one way syncing) the error disappears. Does this mean 
I 
will have problems with setting up replication between these two 
servers? 


[1] 
[@ ~]# doveadm sync -n inbox -u testaccount tcp:mail.local:542 

[2] 
[@ ~]# doveadm sync -n inbox -u testaccount tcp:mail.local:542 
dsync-local(testaccount): Panic: file mbox-lock.c: line 799 
(mbox_lock): 
assertion failed: (lock_type == F_RDLCK || mbox->mbox_lock_type != 
F_RDLCK) 
dsync-local(testaccount): Error: Raw backtrace: 
/usr/lib64/dovecot/libdovecot.so.0(+0xa192e) [0x7fd4f827992e] -> 
/usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) 
[0x7fd4f827999a] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) 
[0x7fd4f8209257] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mbox_lock+0xef) 
[0x7fd4f858feef] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mbox_save_begin+0x658) 
[0x7fd4f8591958] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_save_begin+0x83) 

[0x7fd4f855a573] -> doveadm(+0x4a6a3) [0x55e5ae8276a3] -> 
doveadm(dsync_mailbox_import_mail+0xca) [0x55e5ae82a9fa] -> 
doveadm(dsync_brain_sync_mails+0xb3) [0x55e5ae824963] -> 
doveadm(dsync_brain_run+0x551) [0x55e5ae820551] -> 
doveadm(+0x438e0) 
[0x55e5ae8208e0] -> doveadm(+0x5899f) [0x55e5ae83599f] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
[0x7fd4f828f672] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x1
0f) 
[0x7fd4f8290d5f] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) 
[0x7fd4f828f70c] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7fd4f828f8c8] -> doveadm(+0x297c9) [0x55e5ae8067c9] -> 
doveadm(+0x2bdcc) [0x55e5ae808dcc] -> doveadm(+0x2c9ca) 
[0x55e5ae8099ca] 
-> doveadm(doveadm_mail_try_run+0x215) [0x55e5ae80a325] -> 
doveadm(main+0x46a) [0x55e5ae7f949a] -> 
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd4f7e2c555] -> 
doveadm(+0x1c605) [0x55e5ae7f9605] 
Aborted 


CentOS Linux release 7.8.2003 (Core) 
dovecot-pigeonhole-2.2.36-6.el7_8.1.x86_64 
dovecot-2.2.36-6.el7_8.1.x86_64

Re: DKIM fail if WHM adds Message-ID, should be Message-Id

2020-10-12 Thread @lbutlr 



> On 12 Oct 2020, at 03:06, Tom Hendrikx  wrote:
> 
> On 12-10-2020 10:16, Robert Martin wrote:
>> I created a client library to send emails for a webapp.
>> 
>>  
>> After connecting to the SMTP server with credential setup in CPANEL, and 
>> then do NOT add Message-Id header, the DKIM signature 'h' record created by 
>> dovecot/WHM is wrong, and a Message-ID (with a capital D) header is added, 
>> invalidating the generated DKIM signature value.
>> 
>>  
>> This causes outlook, yahoo, gmail and other email recipients to add 
>> 'dkim:fail' to the message, and thus relegate it to junk or spam.
>> 
>>  
>> The work around is to add to the message a Message-Id with a  little ‘d’ 
>> header.  Then the SMTP server processes the email with the correct generated 
>> DKIM, correct DKIM ‘h’ record and does not add a Message-ID header.
>> 
>>  
>> My SMTP hosting providers that run the WHM/dovecot/CPANEL software are 
>> refusing to raise this as a bug and have requested that I do it.
>> 
> Hi Robert,
> 
> Dovecot does not add DKIM headers, Dovecot doesn't even send email. Aside 
> from that:
> 
> - The RFC states that "Message-ID" is the actual spelling: 
> https://tools.ietf.org/html/rfc5322#section-3.6.4

The RFC (5322) states that the headers are not case sensitive:

"This specification uses the Augmented Backus-Naur Form (ABNF) notation for the 
formal definitions of the syntax of messages.  "

RFC 5234
NOTE:

  ABNF strings are case insensitive and the character set for these
  strings is US-ASCII.

So message-id:, MessAGE-iD:, and Message-Id: are all valid.

-- 
Penny! *Everything* is better with BlueTooth

Doveadm-Sync for sieve / control

2020-10-12 Thread Marc Roos 


How to dsync sieve rules and maybe other necessary files like in 
control?

Re: DKIM fail if WHM adds Message-ID, should be Message-Id

2020-10-12 Thread @lbutlr 
On 12 Oct 2020, at 02:51, webad...@exalt.com.au wrote:
> I created a client library to send emails for a webapp.
> 
> When the client library makes a connection to the SMTP server, and then do 
> NOT add Message-Id header,  Message-ID (with a capital D) header is added, 
> and the SMTP server adds a incorrect DKIM-Signature record 'h' part, 
> invalidating the generated DKIM signature value.

This I s not a dovecot issue I don't think, this is a configuration issue with 
your MTA (exit, postfix, sendmail, et al).

> The work around is to add to the message a Message-Id with a  little ‘d’ 
> header.  Then the SMTP server processes the email with the correct 
> DKIM-Signature, correct DKIM ‘h’ part and does not add a Message-ID header.

This seems very very wrong as the headers in mail messages are not case 
sensitive.

Also, I do not think dovecot adds message-id headers at all, and dovecot would 
not be involved in sending mail.

-- 
Ah we're lonely, we're romantic / and the cider's laced with acid /
and the Holy Spirit's crying, Where's the beef? / And the moon is
swimming naked / and the summer night is fragrant / with a mighty
expectation of relief

Re: DKIM fail if WHM adds Message-ID, should be Message-Id

2020-10-12 Thread Tom Hendrikx 

On 12-10-2020 10:16, Robert Martin wrote:


I created a client library to send emails for a webapp.

After connecting to the SMTP server with credential setup in CPANEL, 
and then do NOT add Message-Id header, the DKIM signature 'h' record 
created by dovecot/WHM is wrong, and a Message-ID (with a capital D) 
header is added, invalidating the generated DKIM signature value.


This causes outlook, yahoo, gmail and other email recipients to add 
'dkim:fail' to the message, and thus relegate it to junk or spam.


The work around is to add to the message a Message-Id with a  little 
‘d’ header.  Then the SMTP server processes the email with the correct 
generated DKIM, correct DKIM ‘h’ record and does not add a Message-ID 
header.


My SMTP hosting providers that run the WHM/dovecot/CPANEL software are 
refusing to raise this as a bug and have requested that I do it.



Hi Robert,

Dovecot does not add DKIM headers, Dovecot doesn't even send email. 
Aside from that:


- The RFC states that "Message-ID" is the actual spelling: 
https://tools.ietf.org/html/rfc5322#section-3.6.4


- The DKIM signer implementation in CPANEL/WHM seems to do the right 
thing: It adds a Message-ID header if it's missing, and creates a DKIM 
signature. It seems unlikely that the CPANEL/WHM DKIM implementation 
generates invalid DKIM signatures for all Message-ID headers it 
generates, as lots of people would have issues with that.


It's unclear from your inital message what's wrong with the generated 
DKIM signature, and it's also unclear where the "Message-Id" header 
(with lowercase 'd') comes from. Could you please explain? Can you show 
full headers for a message with the broken DKIM and all the relevant 
headers?


Kind regards,

Tom

DKIM fail if WHM adds Message-ID, should be Message-Id

2020-10-12 Thread webadmin 



I created a client library to send emails for a webapp.

When the client library makes a connection to the SMTP server, and  
then do NOT add Message-Id header,  Message-ID (with a capital D)  
header is added, and the SMTP server adds a incorrect DKIM-Signature  
record 'h' part, invalidating the generated DKIM signature value.


This causes outlook, yahoo, gmail and other email recipients to add  
'dkim:fail' to the message which pushes it into junk/spam folder.


The work around is to add to the message a Message-Id with a  little  
‘d’ header.  Then the SMTP server processes the email with the correct  
DKIM-Signature, correct DKIM ‘h’ part and does not add a Message-ID  
header.


Message-ID fails DKIM, Message-Id is better by OzBob · Pull Request  
#142 · dovecot/core (github.com)

https://github.com/dovecot/core/pull/142

Regards
Robert

DKIM fail if WHM adds Message-ID, should be Message-Id

2020-10-12 Thread Robert Martin 
I created a client library to send emails for a webapp.

 

After connecting to the SMTP server with credential setup in CPANEL, and
then do NOT add Message-Id header, the DKIM signature 'h' record created by
dovecot/WHM is wrong, and a Message-ID (with a capital D) header is added,
invalidating the generated DKIM signature value.

 

This causes outlook, yahoo, gmail and other email recipients to add
'dkim:fail' to the message, and thus relegate it to junk or spam.

 

The work around is to add to the message a Message-Id with a  little 'd'
header.  Then the SMTP server processes the email with the correct generated
DKIM, correct DKIM 'h' record and does not add a Message-ID header.

 

My SMTP hosting providers that run the WHM/dovecot/CPANEL software are
refusing to raise this as a bug and have requested that I do it.

 

Regards

Robert

Re: Preparing for replication: dsync-local(testaccount): Panic: file mbox-lock.c

2020-10-12 Thread Aki Tuomi 


 
 
  
   Replication is not supported with mbox format. You can only do unidirectional sync out of mbox.
  
  
   
  
  
   Aki
  
  
   
On 12/10/2020 00:25 Marc Roos  wrote:
   
   

   
   

   
   
I am preparing a bit for setting up replication. However when I manually
   
   
try to dsync an account, the first time I execute this[1] command it
   
   
seems to be ok. The 2nd time I am getting this error[2].
   
   
If I add -1 (one way syncing) the error disappears. Does this mean I
   
   
will have problems with setting up replication between these two
   
   
servers?
   
   

   
   

   
   
[1]
   
   
[@ ~]# doveadm sync -n inbox -u testaccount tcp:mail.local:542
   
   

   
   
[2]
   
   
[@ ~]# doveadm sync -n inbox -u testaccount tcp:mail.local:542
   
   
dsync-local(testaccount): Panic: file mbox-lock.c: line 799 (mbox_lock):
   
   
assertion failed: (lock_type == F_RDLCK || mbox->mbox_lock_type !=
   
   
F_RDLCK)
   
   
dsync-local(testaccount): Error: Raw backtrace:
   
   
/usr/lib64/dovecot/libdovecot.so.0(+0xa192e) [0x7fd4f827992e] ->
   
   
/usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x2a)
   
   
[0x7fd4f827999a] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0)
   
   
[0x7fd4f8209257] ->
   
   
/usr/lib64/dovecot/libdovecot-storage.so.0(mbox_lock+0xef)
   
   
[0x7fd4f858feef] ->
   
   
/usr/lib64/dovecot/libdovecot-storage.so.0(mbox_save_begin+0x658)
   
   
[0x7fd4f8591958] ->
   
   
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_save_begin+0x83)
   
   
[0x7fd4f855a573] -> doveadm(+0x4a6a3) [0x55e5ae8276a3] ->
   
   
doveadm(dsync_mailbox_import_mail+0xca) [0x55e5ae82a9fa] ->
   
   
doveadm(dsync_brain_sync_mails+0xb3) [0x55e5ae824963] ->
   
   
doveadm(dsync_brain_run+0x551) [0x55e5ae820551] -> doveadm(+0x438e0)
   
   
[0x55e5ae8208e0] -> doveadm(+0x5899f) [0x55e5ae83599f] ->
   
   
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52)
   
   
[0x7fd4f828f672] ->
   
   
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
   
   
[0x7fd4f8290d5f] ->
   
   
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
   
   
[0x7fd4f828f70c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38)
   
   
[0x7fd4f828f8c8] -> doveadm(+0x297c9) [0x55e5ae8067c9] ->
   
   
doveadm(+0x2bdcc) [0x55e5ae808dcc] -> doveadm(+0x2c9ca) [0x55e5ae8099ca]
   
   
-> doveadm(doveadm_mail_try_run+0x215) [0x55e5ae80a325] ->
   
   
doveadm(main+0x46a) [0x55e5ae7f949a] ->
   
   
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd4f7e2c555] ->
   
   
doveadm(+0x1c605) [0x55e5ae7f9605]
   
   
Aborted
   
   

   
   

   
   
CentOS Linux release 7.8.2003 (Core)
   
   
dovecot-pigeonhole-2.2.36-6.el7_8.1.x86_64
   
   
dovecot-2.2.36-6.el7_8.1.x86_64
   
  
  
   
  
  
   ---
Aki Tuomi