Plugin "last-login" - Log SMTP-AUTH information

[Benjamin Hück]

Hello all,


we are using the Plugin "last-login" [1] to update user's last-login timestamp 
or the timestamp of the last incoming message in a MySQL backend.


Would it be possible to expand the plugin to also log the timestamp of the last 
smtp-auth attempt/successful process?


Currently in our settings, a smtp-auth request (port 587) is forwarded by 
postfix to a dovecot backend server for authentication.


I couldn`t find out the developer of the plugin otherwise I had contact them 
immediatly.


Thanks and best regards,

Benjamin Hück



[1] http://wiki2.dovecot.org/Plugins/LastLogin

--


KEVAG Telekom GmbH
Cusanusstr. 7
D 56073 Koblenz
Fon: +49 261 20162-0
Fax: +49 261 20162-25100
http://www.kevag-telekom.de/

Geschäftsführer:Bernd Gowitzke, Gerd Thewalt
Sitz der Gesellschaft:  Koblenz, Amtsgericht Koblenz, HRB Nr. 5343
USt.IdNr. DE 18 77 67 843   St-Nr. 22/650/0182/7

Re: overview zlib efficiency? Summary and added note

[Haravikk]

Not sure how you’re seeing such a high ratio; I tried the same commands on my 
system (thanks for these btw) and my savings from compression are around 5% =D

That said I’m dealing with a much smaller volume (3gb) and I’ve only identified 
a half dozen or so attachments that don’t have some kind of compression 
already; most modern mail programs will compress common types like images by 
default, and many modern file-formats have compression built in, and can give 
better results than zlib anyway.

My biggest savings are on mailing list messages (I filter these into their own 
mailbox) since they tend to be longer than typical messages, especially with 
auto-quoting, they also tend to be very busy mailboxes, but I also don’t keep 
them forever.

As an experiment I also tried moving my (uncompressed) messages to a 
compressing file-system (ZFS using lz4) but the savings were similarly small; I 
assume they were probably a bit better, but the extra overhead of the 
file-system eroded it since the savings are so small in my case. I think if 
you’re serious about compression then a compressing file-system is the way to 
go though, but in my case I’m on virtual hosting so there’s not much point in 
layering a ZFS volume on top of shared storage (since it’s ZFS based already 
for integrity/redundancy).

I just thought I’d mention my experience since people are quoting big savings 
that I haven’t seen; I wouldn’t consider my usage all that unusual, maybe some 
of you are receiving a lot more newsletter type traffic (these messages can be 
quite large), uncompressed document type files, or are less selective in which 
messages are retained forever? Just a caution that people looking at 
compression may not see the same savings depending upon their actual content.

Spam is another bad category for compression I’ve found; at least in my case 
the messages are usually very short, and/or contain randomised junk to try to 
confound filters, though I’m pretty aggressive about clearing them (I discard 
messages outright above a certain threshold, and use a script to expunge spam 
messages so that I can expunge messages with higher spam ratings faster (so 
possible false positives stick around longer so they can be caught).

> On 16 Mar 2016, at 09:48, Harald Leithner  wrote:
> 
> Hi,
> 
> use "doveadm" to get all real message
> 
> doveadm -f table fetch -A "size.physical" ALL | awk 
> '{s+=$2}END{printf("%.2fMB\n", s/1024/1024);}'
> 
> 189247.67MB .. 185G
> 
> use "du" to get size on disc:
> 
> In my case
> with deduplication:
> 
> /srv/stroage/# du -s -h *
> 53G vmail
> 75G vmail_sis
> 
> without deduplication
> 
> /srv/stroage/# du -s -h -l *
> 53G vmail
> 209Gvmail_sis
> 
> j4i, SIS can't use the zlib plugin so the 75G in my case are not compressed 
> (I haven't a filesystem that I trust and has a compression feature). Anyway 
> it has a 3:1 ratio in my case.
> 
> Maybe I interpret the SIS wrong and SIS couldn't be counted with du -l (count 
> links).
> 
> But if someone doesn't have SIS this values should be point you into the 
> right direction.
> 
> bye
> 
> Harald
> 
> Am 16.03.2016 um 08:50 schrieb Götz Reinicke - IT Koordinator:
>> Am 15.03.16 um 16:01 schrieb Götz Reinicke - IT Koordinator:
>>> Hi,
>>> 
>>> may be someone has already done that: Do you have a script(?) tool which
>>> shows the efficiency of the mail compression if zlib is used?
>>> 
>>> Something that shows the uncompressed size vrs. the compressed.
>> 
>> Hi,
>> 
>> maybe my question was a bit misleading. But anyway thanks for your
>> feedback regarding your experiences and compression rates.
>> 
>> We already thought about the benefit of less IO and more CPU power,
>> which is no concern.
>> 
>> The mailboxes I checked also go with 40-60% compression rate.
>> 
>> But what I was looking for was a tool or way to see what volume would be
>> used if we where not using compression.
>> 
>> e.g. "du -hs --without-zlib"
>> 
>> Our management would like to see a graph one day which shows the volume
>> uncompressed and compressed ...
>> 
>> Adding zlib with mdbox or maildir - as we do it currently - is from my
>> POV if you have the CPU power a MUST :)
>> 
>>  happy dovecoting - Götz
>> 
>> 
>> 
> 
> -- 
> Harald Leithner
> 
> ITronic
> Wiedner Hauptstraße 120/5.1, 1050 Wien, Austria
> Tel: +43-1-545 0 604
> Mobil: +43-699-123 78 4 78
> Mail: leith...@itronic.at | itronic.at

Re: Index corruption in 2.2.18

[Timo Sirainen]

On 16 Mar 2016, at 20:09, Anton Chevychalov  wrote:
> 
> We are using dbox with lz4 compression and dovecot-lda + sieve for delivery.
> 
> Mail recieved:
> Mar 16 06:27:55 mail2 dovecot: lda(s...@example.com): sieve: 
> msgid=<20160316032754.4c89d...@rdbms.oldstable.office.example.com>: stored 
> mail into mailbox 'INBOX'
> 
> Around 08:14:00 user start his thunderbird with local filters. He found one 
> mail without headers.
> 
> Around that time user do repair folder in thunderbird:
> Mar 16 10:14:44 mail2 dovecot: imap(s...@example.com): Error: 
> read(/mail/boxes/example.com/svs/storage/m.139) failed: Cached message size 
> larger than expected (1572 > 980, box=Logs/Backup, UID=9763, cached 
> Message-Id=<20160316032754.4c89d...@rdbms.oldstable.office.example.com>)
> Mar 16 10:14:44 mail2 dovecot: imap(s...@example.com): Error: Corrupted index 
> cache file 
> /mail/boxes/example.com/svs/mailboxes/Logs/Backup/dbox-Mails/dovecot.index.cache:
>  Broken physical size for mail UID 9763
> Mar 16 10:14:44 mail2 dovecot: imap(s...@example.com): Error: 
> read(/mail/boxes/example.com/svs/storage/m.139) failed: Cached message size 
> larger than expected (1572 > 980, box=Logs/Backup, UID=9763, cached 
> Message-Id=<20160316032754.4c89d...@rdbms.oldstable.office.example.com>) 
> (FETCH BODY[] for mailbox Logs/Backup UID 9763)
> Mar 16 10:14:44 mail2 dovecot: imap(s...@example.com): Disconnected: FETCH 
> read() failed in=14812 out=1077049
> Mar 16 10:14:45 mail2 dovecot: imap(s...@example.com): Error: 
> read(/mail/boxes/example.com/svs/storage/m.139): FETCH BODY[] for mailbox 
> Logs/Backup UID 9763 got too little data: 1008 vs 1613
> Mar 16 10:14:45 mail2 dovecot: imap(s...@example.com): Error: Corrupted index 
> cache file 
> /mail/boxes/example.com/svs/mailboxes/Logs/Backup/dbox-Mails/dovecot.index.cache:
>  Broken virtual size for mail UID 9763
> Mar 16 10:14:45 mail2 dovecot: imap(s...@example.com): Disconnected:
> FETCH failed in=837 out=641651
> 
> And still in a same state:
> Mar 16 11:42:51 mail2 dovecot: imap(s...@example.com): Error: 
> read(/mail/boxes/example.com/svs/storage/m.139): FETCH BODY[] for mailbox 
> Logs/Backup UID 9763 got too little data: 1008 vs 1613
> Mar 16 11:42:51 mail2 dovecot: imap(s...@example.com): Error: Corrupted index 
> cache file 
> /mail/boxes/example.com/svs/mailboxes/Logs/Backup/dbox-Mails/dovecot.index.cache:
>  Broken virtual size for mail UID 9763
> Mar 16 11:42:51 mail2 dovecot: imap(s...@example.com): Disconnected: FETCH 
> failed in=12462 out=984043
> 
> In thunderbird we see raw lz4 data.
> 
> I have today backup (duplicate mail on exim to per day mbox). So I can 
> restore that message, but I found older problems in other mailboxes and I 
> can't restore it due to tapes rotation 
> 
> I see a lot of fixes in 2.2.19? Was it fixed after 2.2.18?

Probably this: 
https://github.com/dovecot/core/commit/879dfd4da1a9a379936b8d51c1e00c3bd77873b4

The mails aren't actually corrupted. Dovecot just didn't detect LZ4 compressed 
mails correctly always. v2.2.22 should help.

Re: overview zlib efficiency?

[Teemu Huovila]

On 15.03.2016 21:45, Sven Hartge wrote:
> Robert L Mathews  wrote:
> 
>> Also keep in mind that even if it does increase CPU usage, it reduces
>> disk usage. This is probably an excellent tradeoff for most people,
>> since most servers are limited by disk throughput/latency more than
>> CPU power.
> 
> IOPS are harder to scale (meaning: cost more to scale) than CPU power.
> 
> And gzip (or lz4 of implemented someday) (or even blosc:
liblz4 has been supported since 2.2.11+ http://wiki2.dovecot.org/Plugins/Zlib

> http://www.blosc.org/. They claim "Designed to transmit data to the
> processor cache faster than a memcpy() OS call.") is effectively free
> with todays CPUs.
> 
> Grüße,
> Sven.
> 

Re: overview zlib efficiency? Summary and added note

[Harald Leithner]

Hi,

use "doveadm" to get all real message

doveadm -f table fetch -A "size.physical" ALL | awk 
'{s+=$2}END{printf("%.2fMB\n", s/1024/1024);}'


189247.67MB .. 185G

use "du" to get size on disc:

In my case
with deduplication:

/srv/stroage/# du -s -h *
53G vmail
75G vmail_sis

without deduplication

/srv/stroage/# du -s -h -l *
53G vmail
209Gvmail_sis

j4i, SIS can't use the zlib plugin so the 75G in my case are not 
compressed (I haven't a filesystem that I trust and has a compression 
feature). Anyway it has a 3:1 ratio in my case.


Maybe I interpret the SIS wrong and SIS couldn't be counted with du -l 
(count links).


But if someone doesn't have SIS this values should be point you into the 
right direction.


bye

Harald

Am 16.03.2016 um 08:50 schrieb Götz Reinicke - IT Koordinator:

Am 15.03.16 um 16:01 schrieb Götz Reinicke - IT Koordinator:

Hi,

may be someone has already done that: Do you have a script(?) tool which
shows the efficiency of the mail compression if zlib is used?

Something that shows the uncompressed size vrs. the compressed.


Hi,

maybe my question was a bit misleading. But anyway thanks for your
feedback regarding your experiences and compression rates.

We already thought about the benefit of less IO and more CPU power,
which is no concern.

The mailboxes I checked also go with 40-60% compression rate.

But what I was looking for was a tool or way to see what volume would be
used if we where not using compression.

e.g. "du -hs --without-zlib"

Our management would like to see a graph one day which shows the volume
uncompressed and compressed ...

Adding zlib with mdbox or maildir - as we do it currently - is from my
POV if you have the CPU power a MUST :)

happy dovecoting - Götz





--
Harald Leithner

ITronic
Wiedner Hauptstraße 120/5.1, 1050 Wien, Austria
Tel: +43-1-545 0 604
Mobil: +43-699-123 78 4 78
Mail: leith...@itronic.at | itronic.at

Index corruption in 2.2.18

[Anton Chevychalov]

We are using dbox with lz4 compression and dovecot-lda + sieve for delivery.

Mail recieved:
Mar 16 06:27:55 mail2 dovecot: lda(s...@example.com): sieve: 
msgid=<20160316032754.4c89d...@rdbms.oldstable.office.example.com>: stored mail 
into mailbox 'INBOX'

Around 08:14:00 user start his thunderbird with local filters. He found one 
mail without headers.

Around that time user do repair folder in thunderbird:
Mar 16 10:14:44 mail2 dovecot: imap(s...@example.com): Error: 
read(/mail/boxes/example.com/svs/storage/m.139) failed: Cached message size 
larger than expected (1572 > 980, box=Logs/Backup, UID=9763, cached 
Message-Id=<20160316032754.4c89d...@rdbms.oldstable.office.example.com>)
Mar 16 10:14:44 mail2 dovecot: imap(s...@example.com): Error: Corrupted index 
cache file 
/mail/boxes/example.com/svs/mailboxes/Logs/Backup/dbox-Mails/dovecot.index.cache:
 Broken physical size for mail UID 9763
Mar 16 10:14:44 mail2 dovecot: imap(s...@example.com): Error: 
read(/mail/boxes/example.com/svs/storage/m.139) failed: Cached message size 
larger than expected (1572 > 980, box=Logs/Backup, UID=9763, cached 
Message-Id=<20160316032754.4c89d...@rdbms.oldstable.office.example.com>) (FETCH 
BODY[] for mailbox Logs/Backup UID 9763)
Mar 16 10:14:44 mail2 dovecot: imap(s...@example.com): Disconnected: FETCH 
read() failed in=14812 out=1077049
Mar 16 10:14:45 mail2 dovecot: imap(s...@example.com): Error: 
read(/mail/boxes/example.com/svs/storage/m.139): FETCH BODY[] for mailbox 
Logs/Backup UID 9763 got too little data: 1008 vs 1613
Mar 16 10:14:45 mail2 dovecot: imap(s...@example.com): Error: Corrupted index 
cache file 
/mail/boxes/example.com/svs/mailboxes/Logs/Backup/dbox-Mails/dovecot.index.cache:
 Broken virtual size for mail UID 9763
Mar 16 10:14:45 mail2 dovecot: imap(s...@example.com): Disconnected:
FETCH failed in=837 out=641651

And still in a same state:
Mar 16 11:42:51 mail2 dovecot: imap(s...@example.com): Error: 
read(/mail/boxes/example.com/svs/storage/m.139): FETCH BODY[] for mailbox 
Logs/Backup UID 9763 got too little data: 1008 vs 1613
Mar 16 11:42:51 mail2 dovecot: imap(s...@example.com): Error: Corrupted index 
cache file 
/mail/boxes/example.com/svs/mailboxes/Logs/Backup/dbox-Mails/dovecot.index.cache:
 Broken virtual size for mail UID 9763
Mar 16 11:42:51 mail2 dovecot: imap(s...@example.com): Disconnected: FETCH 
failed in=12462 out=984043

In thunderbird we see raw lz4 data.

I have today backup (duplicate mail on exim to per day mbox). So I can restore 
that message, but I found older problems in other mailboxes and I can't restore 
it due to tapes rotation 

I see a lot of fixes in 2.2.19? Was it fixed after 2.2.18?

-- 
Anton Chevychalov

Re: dsync and ProxyAUTH

[Giovanni Mancuso]

Il 16/03/2016 07:15, Timo Sirainen ha scritto:
> On 15 Mar 2016, at 21:00, Giovanni Mancuso  wrote:
>> Hi,
>>
>> can i use dync to migrate a mailboxes through PROX-AUTH?
>>
>> Could It be an attractive feature for you?
> Do you mean the Sun/Oracle/whatever server's PROXYAUTH command? Already done: 
> https://github.com/dovecot/core/commit/bd06c77a12bb02871b25dceb749fa955f4a272ff
Good News!!! :-) :-)

This feature isn't documented :-) :-)

Thank you very much

v2.2.22 released

[Timo Sirainen]

http://dovecot.org/releases/2.2/dovecot-2.2.22.tar.gz
http://dovecot.org/releases/2.2/dovecot-2.2.22.tar.gz.sig

 + Added doveadm HTTP API: See
   http://wiki2.dovecot.org/Design/DoveadmProtocol/HTTP
 + virtual plugin: Mailbox filtering can now be done based on the
   mailbox metadata. See http://wiki2.dovecot.org/Plugins/Virtual
 + stats: Added doveadm stats reset to reset global stats.
 + stats: Added authentication statistics if auth_stats=yes.
 + dsync, imapc, pop3c & pop3-migration: Many optimizations,
   improvements and error handling fixes.
 + doveadm: Most commands now stop soon after SIGINT/SIGTERM.
 - auth: Auth caching was done too aggressively when %variables were
   used in default_fields, override_fields or LDAP pass/user_attrs.
   userdb result_* were also ignored when user was found from cache.
 - imap: Fixed various assert-crashes caused v2.2.20+. Some of them
   caught actual hangs or otherwise unwanted behavior towards IMAP
   clients.
 - Expunges were forgotten in some situations, for example when
   pipelining multiple IMAP MOVE commands.
 - quota: Per-namespaces quota were broken for dict and count backends
   in v2.2.20+
 - fts-solr: Search queries were using OR instead of AND as the
   separator for multi-token search queries in v2.2.20+.
 - Single instance storage support wasn't really working in v2.2.16+
 - dbox: POP3 message ordering wasn't working correctly.
 - virtual plugin: Fixed crashes related to backend mailbox deletions.

Re: overview zlib efficiency? Summary and added note

[Götz Reinicke - IT Koordinator]

Am 15.03.16 um 16:01 schrieb Götz Reinicke - IT Koordinator:
> Hi,
> 
> may be someone has already done that: Do you have a script(?) tool which
> shows the efficiency of the mail compression if zlib is used?
> 
> Something that shows the uncompressed size vrs. the compressed.

Hi,

maybe my question was a bit misleading. But anyway thanks for your
feedback regarding your experiences and compression rates.

We already thought about the benefit of less IO and more CPU power,
which is no concern.

The mailboxes I checked also go with 40-60% compression rate.

But what I was looking for was a tool or way to see what volume would be
used if we where not using compression.

e.g. "du -hs --without-zlib"

Our management would like to see a graph one day which shows the volume
uncompressed and compressed ...

Adding zlib with mdbox or maildir - as we do it currently - is from my
POV if you have the CPU power a MUST :)

happy dovecoting - Götz





smime.p7s
Description: S/MIME Cryptographic Signature

Re: New dovecot user needs installation help

[Steffen Kaiser]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, 15 Mar 2016, Jason Pruim wrote:

On Mar 15, 2016, at 8:21 AM, Steffen Kaiser  
wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, 15 Mar 2016, Jason Pruim wrote:

On Mar 15, 2016, at 3:22 AM, Steffen Kaiser  
wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


On Mon, 14 Mar 2016, Jason Pruim wrote:

So I just setup my first dovecot install with postfix running for the mail 
server… I can telnet into my dovecot install and login just fine


what does "telnet into my dovecot install" mean exacly?


I used telnet to connect to port 143 and issued some smtp commands including 
logging into an account.


I issued SMTP commands on port 143 ?


telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE 
STARTTLS AUTH=PLAIN] Dovecot ready.
a login "beth" "beth4338"
a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT 
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN 
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT 
SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS] Logged in
select inbox
select BAD Error in IMAP command INBOX: Unknown command.


add any word before the select, too, e.g.:

a select inbox

OK, it's working from inside, now:

telnet external-IP 143
from the same host, to verify that Dovecot listens on the external 
interface, then try from outside ...



Did you've read:
http://wiki2.dovecot.org/TestInstallation


I did yes... But I will go back over it tonight when I get back home.



with my user locally… But when I try to add the account from my phone, 
computer, or telnetting from my local computer, it just hangs and eventually 
kicks back a generic error saying it can’t be added… Where can I start looking? 
I’m hosted with amazon web services and I’m waiting on them to approve my 
request to white list my IP on my instance, but that should only restrict my 
ability to send email out on port 25… I should still be able to operate locally…


According to the firewall on amazon port 143 and 993 are both allowed from any 
IP address into the server (All outbound traffic is currently open until I get 
it working)

I’m thinking it’s something easy, and I can handle pointers! I don’t 
necessarily need my hand held the entire way… At least not yet…

Here is the output of my dovecot -n:

[ec2-user@ip-172-31-22-222 log]$ dovecot -n
# 2.0.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.14.48-33.39.amzn1.x86_64 x86_64
mail_location = mbox:~/mail:INBOX=/var/mail/%u
mbox_write_locks = fcntl
passdb {
driver = pam
}
ssl_cert = 

- -- Steffen Kaiser
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEVAwUBVuf+Ynz1H7kL/d9rAQLCAQgAy/MZhOYJHjtIYK3L7149W7UiXtn50WBR
4bq9322YfcxtCWeGGsVdz5q/IM7Sz1WnQR5tFGd3LhaMb/g5FjOUzMGZiAaz3Qwu
AcrC+VVYT1cDhXj/7IMRnKvWWH1l0caSU54rGiKrp5lWKOqmGDabxwYEKFnV8aRC
Y3RXu+ZQSVaDjHxnwG8ydFGdRTjkJ5w4GndJG5xPJm6W+56QjEgXgerTeQkCx3vf
mMHqg3BqJ5jWC1cVmnix3kMSpZvlJDRLmc/YIv7Ycvkic+QPisO27b5Wmmgb79GH
OtOTiotR7thlO/ktP16OLuj8oGPVislBcXDfliyDYkLEg6koRbuZWg==
=fu00
-END PGP SIGNATURE-





- -- 
Steffen Kaiser

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEVAwUBVukBDXz1H7kL/d9rAQLUngf9EmUTTFxcRUcO73Ugr6h2dnZQghTSzezh
WptjOCtMgA1FeOLnQ4bysOh7RYF7poo6Keg8fyDqNkn5rdGz7kHugiZWKx4u9yf7
AnZXQKIX9hSm6fRsTqAO+2RJ14VpDw4h0EP9CwNfDNf/e2EWyInCAf2klGCIGMh+
AktSg7DEjbZtkotflXjeJCBfFTpAAMfLtfVi72cwxn2LrvNUvxjnGO4sQz4Evgmk
xLQXVtqqjdOP1HEmJIRB4pNEpl9aglbd385EHOOqXyDAe7A9t9oZ6/9K9IWO/HEY
eLtvCpK02yUrDJzYIx5L9B/Nrpn5AOZ7yK4alawyGvdXGyOH88jiBA==
=W1+W
-END PGP SIGNATURE-

Re: dsync and ProxyAUTH

[Timo Sirainen]

On 15 Mar 2016, at 21:00, Giovanni Mancuso  wrote:
> 
> Hi,
> 
> can i use dync to migrate a mailboxes through PROX-AUTH?
> 
> Could It be an attractive feature for you?

Do you mean the Sun/Oracle/whatever server's PROXYAUTH command? Already done: 
https://github.com/dovecot/core/commit/bd06c77a12bb02871b25dceb749fa955f4a272ff