Re: two unrelated issues, lastlogin, and an out of memory fatal error

[Aki Tuomi]

Hi!


On 10.04.2018 19:31, David Mehler wrote:
> Hello,
>
> I've got a few misc issues that I'm trying to get a handle on.
>
> I'm having what is probably a simple error with last logins. It's not
> working, unable to write values to the database. Here's the log lines:
>
> 2018-04-10 10:43:34 imap-login: Info: Login: user=,
> method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=75934, TLS,
> session=
> 2018-04-10 10:43:34 imap(u...@domain.com)<75934>:
> Error: last_login_dict: Failed to write value for user
> u...@domain.com: dict-server returned failure: sql dict: commit
> failed: Field 'name' doesn't have a default value (reply took 0.013
> secs (0.000 in dict wait, 0.012 in other ioloops, 0.000 in locks,
> async-id reply 0.002 secs ago, started on dict-server 0.005 secs ago,
> took 0.005 secs))
It's attempting to insert a new row there for some reason. I'd recommend
using separate table for last-login values.

> Here's an excerpt from my dovecot-dict-sql.conf.ext file:
>
> map {
>   pattern = shared/last-login/$user
>   table = accounts
>   value_field = last_login
>   value_type = uint
>
>   fields {
> username = $user
>   }
> }
>
> and the relevant database table:
> describe accounts;
> ++--+--+-+-++
>
> | Field  | Type | Null | Key | Default | Extra  |
>
> ++--+--+-+-++
>
> | id | int(10) unsigned | NO   | PRI | NULL| auto_increment |
>
> | name   | varchar(255) | NO   | | NULL||
>
> | username   | varchar(64)  | NO   | MUL | NULL||
>
> | domain | varchar(255) | NO   | MUL | NULL||
>
> | password   | varchar(255) | NO   | | NULL||
>
> | quota  | int(10) unsigned | YES  | | 0   ||
>
> | enabled| tinyint(1)   | YES  | | 0   ||
>
> | sendonly   | tinyint(1)   | YES  | | 0   ||
>
> | last_login | int(11)  | YES  | | NULL||
>
>
> Second issue, I've used imapsync to download my gmail messages to
> dovecot. I am atempting to access these messages and getting:
> 2018-04-08 11:11:58
> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:StoZDD4xylqhrAAA0J78UA>:
> Info: Indexed 0 messages in INBOX
> 2018-04-08 11:11:58
> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:0AaXFD4xylqhrAAA0J78UA>:
> Info: Indexed 0 messages in INBOX
> 2018-04-08 11:11:58
> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:CP3TIT4xylqhrAAA0J78UA>:
> Info: Indexed 0 messages in INBOX
> 2018-04-08 11:11:58
> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:eNEWMT4xylqhrAAA0J78UA>:
> Info: Indexed 0 messages in INBOX
> 2018-04-08 11:15:28 imap(u...@domain.com)<21374><4GzoqFdpisJ/AAAB>:
> Fatal: block_alloc(8388608): Out of memory
> 2018-04-08 11:15:28 imap(u...@domain.com)<21374><4GzoqFdpisJ/AAAB>:
> Fatal: master: service(imap): child 21374 returned error 83 (Out of
> memory (service imap { vsz_limit=256 MB }, you may need to increase
> it) - set CORE_OUTOFMEM=1 environment to get core dump)
>
> when I atempt to do so. For completeness the imapsync command I used was:
>
> imapsync --host1 imap.gmail.com --user1 Username --password1 Password
> --host2 localhost --user2 username --password2 Password --ssl1
> --useheader 'Message-Id' --regextrans2 's/\[Gmail\]/user\@domain/'
>

Increase memory, your cache takes more memory than you have anticipated.

> Any help appreciated on either issue.
> Thanks.
> Dave.
>
> doveconf -n
> # 2.3.1 (8e2f634): /usr/local/etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.1 (d9bc6dfe)
> # OS: FreeBSD 11.1-RELEASE-p4 amd64
> # Hostname: localhost
> auth_cache_size = 24 M
> auth_cache_ttl = 18 hours
> auth_default_realm = domain.com
> auth_mechanisms = plain login
> dict {
>   acl = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext
>   lastlogin = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext
>   quota = mysql:/usr/local/etc/dovecot/dovecot-dict-sql.conf.ext
> }
> first_valid_gid = 999
> first_valid_uid = 999
> hostname = ..
> imap_idle_notify_interval = 10 mins
> last_valid_gid = 999
> last_valid_uid = 999
> lda_mailbox_autocreate = yes
> lda_mailbox_autosubscribe = yes
> lda_original_recipient_header = X-Original-To
> listen = 127.0.0.1 xxx.xxx.xxx.xxx
> log_path = /var/log/dovecot/dovecot.log
> log_timestamp = "%Y-%m-%d %H:%M:%S "
> mail_access_groups = vmail
> mail_gid = vmail
> mail_home = /home/vmail/mailboxes/%d/%n
> mail_location = maildir:~/mail:LAYOUT=fs
> mail_plugins = acl mail_log notify quota trash virtual welcome zlib
> mail_privileged_group = vmail
> mail_server_admin = mailto:postmas...@davemehler.com
> mail_uid = vmail
> mailbox_idle_check_interval = 59 secs
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto 

Re: two unrelated issues, lastlogin, and an out of memory fatal error

[Thomas Zajic]

* David Mehler, 2018-04-11 17:23

> [...]
> The issue is the 1523459718  I was expecting something like a time
> stamp. Is this fixable? Also, can I use last_login to see on which IP
> the user last logged in from?
> [...]

This is in fact a timestamp:

[zlatko@disclosure:~]$ date -d @1523459718
Mit Apr 11 17:15:18 CEST 2018

Your output might look different depending on your locale.

HTH,
Thomas

Re: two unrelated issues, lastlogin, and an out of memory fatal error

[Reio Remma]

On 11.04.18 18:23, David Mehler wrote:

Hello Aki,

Thanks for your suggestion. I followed it and the wiki at:

https://wiki.dovecot.org/Plugins/LastLogin

Good news, I get a record in the users table, with the username that
just logged in. This is good. Bad news is I got unexpected output from
the last_login time stamp. Here's the table and the user:

mysql> describe users;
++--+--+-+-+---+
| Field  | Type | Null | Key | Default | Extra |
++--+--+-+-+---+
| userid | varchar(255) | NO   | PRI | NULL|   |
| last_login | int(11)  | YES  | | NULL|   |
++--+--+-+-+---+
2 rows in set (0.00 sec)

mysql> select * from users;
+++
| userid | last_login |
+++
| u...@domain.com | 1523459718 |
+++
1 row in set (0.01 sec)


The issue is the 1523459718  I was expecting something like a time
stamp. Is this fixable? Also, can I use last_login to see on which IP
the user last logged in from?


SELECT FROM_UNIXTIME(1523459718)

Maybe that will help?

Good luck,
Reio

Re: two unrelated issues, lastlogin, and an out of memory fatal error

[David Mehler]

Hello Aki,

Thanks for your suggestion. I followed it and the wiki at:

https://wiki.dovecot.org/Plugins/LastLogin

Good news, I get a record in the users table, with the username that
just logged in. This is good. Bad news is I got unexpected output from
the last_login time stamp. Here's the table and the user:

mysql> describe users;
++--+--+-+-+---+
| Field  | Type | Null | Key | Default | Extra |
++--+--+-+-+---+
| userid | varchar(255) | NO   | PRI | NULL|   |
| last_login | int(11)  | YES  | | NULL|   |
++--+--+-+-+---+
2 rows in set (0.00 sec)

mysql> select * from users;
+++
| userid | last_login |
+++
| u...@domain.com | 1523459718 |
+++
1 row in set (0.01 sec)


The issue is the 1523459718  I was expecting something like a time
stamp. Is this fixable? Also, can I use last_login to see on which IP
the user last logged in from?

Any ideas on my other issue, the one on the running out of memory when
attempting to access the imported gmail messages?

Thanks for your help so far.

Thanks.
Dave.


On 4/11/18, Aki Tuomi  wrote:
> Hi!
>
>
> On 10.04.2018 19:31, David Mehler wrote:
>> Hello,
>>
>> I've got a few misc issues that I'm trying to get a handle on.
>>
>> I'm having what is probably a simple error with last logins. It's not
>> working, unable to write values to the database. Here's the log lines:
>>
>> 2018-04-10 10:43:34 imap-login: Info: Login: user=,
>> method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=75934, TLS,
>> session=
>> 2018-04-10 10:43:34 imap(u...@domain.com)<75934>:
>> Error: last_login_dict: Failed to write value for user
>> u...@domain.com: dict-server returned failure: sql dict: commit
>> failed: Field 'name' doesn't have a default value (reply took 0.013
>> secs (0.000 in dict wait, 0.012 in other ioloops, 0.000 in locks,
>> async-id reply 0.002 secs ago, started on dict-server 0.005 secs ago,
>> took 0.005 secs))
> It's attempting to insert a new row there for some reason. I'd recommend
> using separate table for last-login values.
>
>> Here's an excerpt from my dovecot-dict-sql.conf.ext file:
>>
>> map {
>>   pattern = shared/last-login/$user
>>   table = accounts
>>   value_field = last_login
>>   value_type = uint
>>
>>   fields {
>> username = $user
>>   }
>> }
>>
>> and the relevant database table:
>> describe accounts;
>> ++--+--+-+-++
>>
>> | Field  | Type | Null | Key | Default | Extra  |
>>
>> ++--+--+-+-++
>>
>> | id | int(10) unsigned | NO   | PRI | NULL| auto_increment |
>>
>> | name   | varchar(255) | NO   | | NULL||
>>
>> | username   | varchar(64)  | NO   | MUL | NULL||
>>
>> | domain | varchar(255) | NO   | MUL | NULL||
>>
>> | password   | varchar(255) | NO   | | NULL||
>>
>> | quota  | int(10) unsigned | YES  | | 0   ||
>>
>> | enabled| tinyint(1)   | YES  | | 0   ||
>>
>> | sendonly   | tinyint(1)   | YES  | | 0   ||
>>
>> | last_login | int(11)  | YES  | | NULL||
>>
>>
>> Second issue, I've used imapsync to download my gmail messages to
>> dovecot. I am atempting to access these messages and getting:
>> 2018-04-08 11:11:58
>> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:StoZDD4xylqhrAAA0J78UA>:
>> Info: Indexed 0 messages in INBOX
>> 2018-04-08 11:11:58
>> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:0AaXFD4xylqhrAAA0J78UA>:
>> Info: Indexed 0 messages in INBOX
>> 2018-04-08 11:11:58
>> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:CP3TIT4xylqhrAAA0J78UA>:
>> Info: Indexed 0 messages in INBOX
>> 2018-04-08 11:11:58
>> indexer-worker(u...@domain.com)<44193><4GzoqFdpisJ/AAAB:eNEWMT4xylqhrAAA0J78UA>:
>> Info: Indexed 0 messages in INBOX
>> 2018-04-08 11:15:28 imap(u...@domain.com)<21374><4GzoqFdpisJ/AAAB>:
>> Fatal: block_alloc(8388608): Out of memory
>> 2018-04-08 11:15:28 imap(u...@domain.com)<21374><4GzoqFdpisJ/AAAB>:
>> Fatal: master: service(imap): child 21374 returned error 83 (Out of
>> memory (service imap { vsz_limit=256 MB }, you may need to increase
>> it) - set CORE_OUTOFMEM=1 environment to get core dump)
>>
>> when I atempt to do so. For completeness the imapsync command I used was:
>>
>> imapsync --host1 imap.gmail.com --user1 Username --password1 Password
>> --host2 localhost --user2 username --password2 Password --ssl1
>> --useheader 'Message-Id' --regextrans2 's/\[Gmail\]/user\@domain/'
>>
>
> 

Re: lda fails in parse_angle_addr if sieve is enabled

[Bernard Spil]

Hi Stephan,

Shared the message to you in person only via separate mail.

With to Return-Path, I've not seen any difference in failures, all
messages were consistently failing with the same error.

Thank you! Bernard.

2018-04-10 2:54 GMT+02:00 Stephan Bosch :
> Op 4/8/2018 om 8:10 PM schreef Bernard Spil:
>> Hi,
>>
>> Since updating to 2.3.1 on my FreeBSD mailserver mail delivery using
>> lda is broken if I have sieve enabled.
>> (Before updating this was 2.2 and pigeonhole 0.4)
>>
>> FreeBSD 11.1-p8 amd64
>> Dovecot 2.3.1
>> Pigeonhole 0.5.1
>>
>> Mailflow is OpenSMTPd as MTA, using mda delivery to rspamc which
>> utlimately delivers using dovecot-lda.
>>
>> smtpd.conf
>> deliver to mda "rspamc -h scan --mime -e
>> \"/usr/local/libexec/dovecot/deliver -d %{user.username}\""
>> %{user.username} is the local user after virtusers, aliases etc.
>> verified using a shell wrapper and capturing the username.
>>
>> conf.d/15-lda.conf
>> protocol lda {
>>   mail_plugins = $mail_plugins sieve
>> }
>>
>>
>> Mail gets delivered.
>>
>> Don't understand why it is looking for a <>-address if sieve is enabled.
>
> I found the cause of the panic. However, what actually triggers it could
> be a little more nasty. Do you have an example of a message passed to
> dovecot-lda that causes the problem? I am particularly interested in the
> Return-Path header.
>
> Regards,
>
> Stephan.
>

Re: lda fails in parse_angle_addr if sieve is enabled

[Stephan Bosch]

Op 11-4-2018 om 20:42 schreef Bernard Spil:

Hi Stephan,

Shared the message to you in person only via separate mail.

With to Return-Path, I've not seen any difference in failures, all
messages were consistently failing with the same error.

Thank you! Bernard.


The fix for the panic itself is here:

https://github.com/dovecot/core/commit/fbed9168dc3b104b09bd748409aec902328cd8e5

Given that you provided an example message without a return-path header, 
I am confused on how this panic would occur at your end. Are you really 
sure this is the message that Dovecot gets to see? If you extracted it 
from the STMP queue, that may not be the same.


You could also obtain a GDB backtrace from dovecot-lda, which would tell 
me from which part of the code this panic is reached (I currently see 
only one obvious path). You can run dovecot-lda from command line within 
GDB to achieve this: `gdb --args dovecot-lda -p  -d 
` and then execute `r` .. wait until it crashes an then issue 'bt 
full` (https://dovecot.org/bugreport.html).


Regards,

Stephan.





2018-04-10 2:54 GMT+02:00 Stephan Bosch :

Op 4/8/2018 om 8:10 PM schreef Bernard Spil:

Hi,

Since updating to 2.3.1 on my FreeBSD mailserver mail delivery using
lda is broken if I have sieve enabled.
(Before updating this was 2.2 and pigeonhole 0.4)

FreeBSD 11.1-p8 amd64
Dovecot 2.3.1
Pigeonhole 0.5.1

Mailflow is OpenSMTPd as MTA, using mda delivery to rspamc which
utlimately delivers using dovecot-lda.

smtpd.conf
deliver to mda "rspamc -h scan --mime -e
\"/usr/local/libexec/dovecot/deliver -d %{user.username}\""
%{user.username} is the local user after virtusers, aliases etc.
verified using a shell wrapper and capturing the username.

conf.d/15-lda.conf
protocol lda {
   mail_plugins = $mail_plugins sieve
}


Mail gets delivered.

Don't understand why it is looking for a <>-address if sieve is enabled.

I found the cause of the panic. However, what actually triggers it could
be a little more nasty. Do you have an example of a message passed to
dovecot-lda that causes the problem? I am particularly interested in the
Return-Path header.

Regards,

Stephan.

dovecot sieve duplicates detection

[André Rodier]

Hello,

I have tested the sieve duplicate script with success so far, but I have
a question.

I would like to know if the "duplicate" sieve flag in Dovecot is global
to all folders, or specific to one folder only.

For instance, if I copy an email from one folder to another, and I have
a discard action on duplicate email, is this action will be applied (in
this case, discard) or not.

If the duplicate is global to all folders, is there a way to restrict
the search in one folder only.

Thanks for your help.
André