from:"Ed W"


On 13/04/2012 12:51, Timo Sirainen wrote:

- Use the checksums to assist with replication speed/efficiency (dsync or 
custom imap commands)

It would be of some use with dbox index rebuilding. I don't think it would help 
with dsync.

..

- File RFCs for new imap features along the lemonde lines which allow clients 
to have faster recovery from corrupted offline states...

Too much trouble, no one would implement it :)


I presume you have seen that cyrus is working on various distributed 
options?  Standardising this through imap might work if they also buy 
into it?




- Storage backends where emails are redundantly stored and might not ALL be on 
a single server (find me the closest copy of email X) - derivations of this 
might be interesting for compliance archiving of messages?
- Fancy key-value storage backends might use checksums as part of the key value 
(either for the whole or parts of the message)

GUID would work for these as well, without the possibility of a hash collision.


I was thinking that the win for key-value store as a backend is if you 
can reduce the storage requirements or do better placement of the data 
(mail text replicated widely, attachments stored on higher latency 
storage?).  Hence whilst I don't see this being a win with current 
options, if it were done then it would almost certainly be per mime 
part, eg storing all large attachments in one place and the rest of the 
message somewhere else, perhaps with different redundancy levels per type


OK, this is all completely pie in the sky.  Please don't build it!  All 
I meant was that these are the kind of things that someone might one day 
desire to do and hence they would have competing requirements for what 
to checksum...


Cheers

Ed W

Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?


On 13/04/2012 13:21, Timo Sirainen wrote:

On 13.4.2012, at 15.17, Ed W wrote:


On 13/04/2012 12:51, Timo Sirainen wrote:

- Use the checksums to assist with replication speed/efficiency (dsync or 
custom imap commands)

It would be of some use with dbox index rebuilding. I don't think it would help 
with dsync.

..

- File RFCs for new imap features along the lemonde lines which allow clients 
to have faster recovery from corrupted offline states...

Too much trouble, no one would implement it :)

I presume you have seen that cyrus is working on various distributed options?  
Standardising this through imap might work if they also buy into it?

Probably more trouble than worth. I doubt anyone would want to run a 
cross-Dovecot/Cyrus cluster.


No definitely not.  Sorry I just meant that you are both working on 
similar things.  Standardising the basics that each use might be useful 
in the future



That can almost be done already .. the attachments are saved and accessed via a 
lib-fs API. It wouldn't be difficult to write a backend for some key-value 
databases. So with about one day's coding you could already have Dovecot save 
all message attachments to a key-value db, and you can configure redundancy in 
the db's configs.


Hmm, super.

Ed W

Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?


On 13/04/2012 06:29, Stan Hoeppner wrote:

On 4/12/2012 5:58 AM, Ed W wrote:


The claim by ZFS/BTRFS authors and others is that data silently bit
rots on it's own. The claim is therefore that you can have a raid1 pair
where neither drive reports a hardware failure, but each gives you
different data?

You need to read those articles again very carefully.  If you don't
understand what they mean by 1 in 10^15 bits non-recoverable read error
rate and combined probability, let me know.


OK, I'll bite.  I only have an honours degree in mathematics from a well 
known university, so grateful if you could dumb it down appropriately?


Lets start with what those articles are you referring to?  I don't see 
any articles if I go literally up the chain from this email, but you 
might be talking about any one of the lots of other emails in this 
thread or even some other email thread?


Wikipedia has it's faults, but it dumbs the silent corruption claim 
down to:

http://en.wikipedia.org/wiki/ZFS
an undetected error for every 67TB

And a CERN study apparently claims far higher than one in every 10^16 bits

Now, I'm NOT professing any experience of axe to grind here.  I'm simply 
asking by what feature do you believe either software or hardware RAID1 
is capable of detecting which pair is correct when both pairs of a raid 
one disk return different results and there is no hardware failure to 
clue us that one pair suffered a read error?  Please don't respond with 
a maths pissing competition, it's an innocent question about what levels 
of data checking are done on each piece of the hardware chain?  My 
(probably flawed) understanding is that popular RAID 1 implementations 
don't add any additional sector checksums over and above what the 
drives/filesystem/etc add already offer - is this the case?





And this has zero bearing on RAID1.  And RAID1 reads don't work the way
you describe above.  I explained this in some detail recently.


Where?



Been working that way for more than 2 decades Ed. :)  Note that RAID1
has that 1 for a reason.  It was the first RAID level.


What should I make of RAID0 then?

Incidentally do you disagree with the history of RAID evolution on 
Wikipedia?

http://en.wikipedia.org/wiki/RAID


Regards

Ed W

Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?

 there exists hardware on sale with *known* defects.  
Despite that the industry continues without collapse.  Now you claim 
that if corruption is silent and people only tend to notice it much 
later and under certain edge conditions that this can't be possible 
because it should cause the industry to collapse..???


...Not buying your logic...

Ed W

Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?

2012-04-12 Thread Ed W


On 12/04/2012 11:20, Stan Hoeppner wrote:

On 4/11/2012 9:23 PM, Emmanuel Noobadmin wrote:

On 4/12/12, Stan Hoeppners...@hardwarefreak.com  wrote:

On 4/11/2012 11:50 AM, Ed W wrote:

One of the snags of md RAID1 vs RAID6 is the lack of checksumming in the
event of bad blocks.  (I'm not sure what actually happens when md
scrubbing finds a bad sector with raid1..?).  For low performance
requirements I have become paranoid and been using RAID6 vs RAID10,
filesystems with sector checksums seem attractive...

Except we're using hardware RAID1 here and mdraid linear.  Thus the
controller takes care of sector integrity.  RAID6 yields nothing over
RAID10, except lower performance, and more usable space if more than 4
drives are used.

How would the control ensure sector integrity unless it is writing
additional checksum information to disk? I thought only a few
filesystems like ZFS does the sector checksum to detect if any data
corruption occurred. I suppose the controller could throw an error if
the two drives returned data that didn't agree with each other but it
wouldn't know which is the accurate copy but that wouldn't protect the
integrity of the data, at least not directly without additional human
intervention I would think.

When a drive starts throwing uncorrectable read errors, the controller
faults the drive and tells you to replace it.  Good hardware RAID
controllers are notorious for their penchant to kick drives that would
continue to work just fine in mdraid or as a single drive for many more
years.  The mindset here is that anyone would rather spent $150-$2500
dollars on a replacement drive than take a chance with his/her valuable
data.



I'm asking a subtlely different question.

The claim by ZFS/BTRFS authors and others is that data silently bit 
rots on it's own. The claim is therefore that you can have a raid1 pair 
where neither drive reports a hardware failure, but each gives you 
different data?  I can't personally claim to have observed this, so it 
remains someone else's theory...  (for background my experience is 
simply: RAID10 for high performance arrays and RAID6 for all my personal 
data - I intend to investigate your linear raid idea in the future though)


I do agree that if one drive reports a read error, then it's quite easy 
to guess which pair of the array is wrong...


Just as an aside, I don't have a lot of failure experience.  However, 
the few I have had (perhaps 6-8 events now) is that there is a massive 
correlation in failure time with RAID1, eg one pair I had lasted perhaps 
2 years and then both failed within 6 hours of each other. I also had a 
bad experience with RAID 5 that wasn't being scrubbed regularly and when 
one drive started reporting errors (ie lack of monitoring meant it had 
been bad for a while), the rest of the array turned out to be a 
patchwork of read errors - linux raid then turns out to be quite fragile 
in the presence of a small number of read failures and it's extremely 
difficult to salvage the 99% of the array which is ok due to the disks 
getting kicked out... (of course regular scrubs would have prevented 
getting so deep into that situation - it was a small cheap nas box 
without such features)


Ed W

Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?

2012-04-12 Thread Ed W

On 12/04/2012 02:18, Stan Hoeppner wrote:

On 4/11/2012 11:50 AM, Ed W wrote:

Re XFS. Have you been watching BTRFS recently?

I will concede that despite the authors considering it production ready
I won't be using it for my servers just yet. However, it's benchmarking
on single disk benchmarks fairly similarly to XFS and in certain cases
(multi-threaded performance) can be somewhat better. I haven't yet seen
any benchmarks on larger disk arrays yet, eg 6+ disks, so no idea how it
scales up. Basically what I have seen seems competitive

Links?

http://btrfs.ipv5.de/index.php?title=Main_Page#Benchmarking

See the regular Phoronix benchmarks in particular. However, I believe
these are all single disk?

I don't have such hardware spare to benchmark, but I would be interested
to hear from someone who benchmarks your RAID1+linear+XFS suggestion,
especially if they have compared a cutting edge btrfs kernel on the same
array?

http://btrfs.boxacle.net/repository/raid/history/History_Mail_server_simulation._num_threads=128.html

This is with an 8 wide LVM stripe over 8 17 drive hardware RAID0 arrays.
If the disks had been setup as a concat of 68 RAID1 pairs, XFS would
have turned in numbers significantly higher, anywhere from a 100%
increase to 500%.

My instinct is that this is an irrelevant benchmark for BTRFS because
its performance characteristics for these workloads have changed so
significantly? I would be far more interested in a 3.2 and then a
3.6/3.7 benchmark in a years time

In particular recent benchmarks on Phoronix show btrfs exceeding XFS
performance on heavily threaded benchmarks - however, I doubt this is
representative of performance on a multi-disk benchmark?

It would be nice to see these folks update these
results with a 3.2.6 kernel, as both BTRFS and XFS have improved
significantly since 2.6.35. EXT4 and JFS have seen little performance
work since.

My understanding is that there was a significant multi-thread
performance boost for EXT4 in the last year kind of timeframe? I don't
have a link to hand, but someone did some work to reduce lock contention
(??) which I seem to recall made a very large difference on multi-user
or multi-cpu workloads? I seem to recall that the summary was that it
allowed Ext4 to scale up to a good fraction of XFS performance on
medium sized systems? (I believe that XFS still continues to scale far
better than anything else on large systems)

Point is that I think it's a bit unfair to say that little has changed
on Ext4? It still seems to be developing faster than maintenance only

However, well OT... The original question was: anyone tried very recent
BTRFS on a multi-disk system. Seems like the answer is no. My proposal
is that it may be worth watching in the future

Cheers

Ed W

P.S. I have always been intrigued by the idea that a COW based
filesystem could potentially implement much faster RAID parity,
because it can avoid reading the whole stripe. The idea is that you
treat unallocated space as zero, which means you can compute the
incremental parity with only a read/write of the checksum value (and
with a COW filesystem you only ever update by rewriting to new zero'd
space). I had in mind something like a fixed parity disk (RAID4?) and
allowing the parity disk to be write behind cached in ram (ie exposed
to risk of: power fails AND data disk fails at the same time). My code
may not be following along for a while though...

Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?

2012-04-12 Thread Ed W


On 12/04/2012 12:09, Timo Sirainen wrote:

On 12.4.2012, at 13.58, Ed W wrote:


The claim by ZFS/BTRFS authors and others is that data silently bit rots on 
it's own. The claim is therefore that you can have a raid1 pair where neither drive 
reports a hardware failure, but each gives you different data?

That's one reason why I planned on adding a checksum to each message in dbox. 
But I forgot to actually do that. I guess I could add it for new messages in 
some upcoming version. Then Dovecot could optionally verify the checksum before 
returning the message to client, and if it detects corruption perhaps 
automatically read it from some alternative location (e.g. if dsync replication 
is enabled ask from another replica). And Dovecot index files really should 
have had some small (8/16/32bit) checksums of stuff as well..



I have to say - I haven't actually seen this happen... Do any of your 
big mailstore contacts observe this, eg rackspace, etc?


I think it's worth thinking about the failure cases before implementing 
something to be honest?  Just sticking in a checksum possibly doesn't 
help anyone unless it's on the right stuff and in the right place?


Off the top of my head:
- Someone butchers the file on disk (disk error or someone edits it with vi)
- Restore of some files goes subtly wrong, eg tool tries to be clever 
and fails, snapshot taken mid-write, etc?

- Filesystem crash (sudden power loss), how to deal with partial writes?


Things I might like to do *if* there were some suitable checksums 
available:
- Use the checksum as some kind of guid either for the whole message, 
the message minus the headers, or individual mime sections
- Use the checksums to assist with replication speed/efficiency (dsync 
or custom imap commands)
- File RFCs for new imap features along the lemonde lines which allow 
clients to have faster recovery from corrupted offline states...
- Single instance storage (presumably already done, and of course this 
has some subtleties in the face of deliberate attack)
- Possibly duplicate email suppression (but really this is an LDA 
problem...)
- Storage backends where emails are redundantly stored and might not ALL 
be on a single server (find me the closest copy of email X) - 
derivations of this might be interesting for compliance archiving of 
messages?
- Fancy key-value storage backends might use checksums as part of the 
key value (either for the whole or parts of the message)


The mail server has always looked like a kind of key-value store to my 
eye.  However, traditional key-value isn't usually optimised for 
streaming reads, hence dovecot seems like a key value store, 
optimised for sequential high speed streaming access to the key 
values...  Whilst it seems increasingly unlikely that a traditional 
key-value store will work well to replace say mdbox, I wonder if it's 
not worth looking at the replication strategies of key-value stores to 
see if those ideas couldn't lead to new features for mdbox?


Cheers

Ed W

Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?

2012-04-11 Thread Ed W


Re XFS.  Have you been watching BTRFS recently?

I will concede that despite the authors considering it production ready 
I won't be using it for my servers just yet.  However, it's benchmarking 
on single disk benchmarks fairly similarly to XFS and in certain cases 
(multi-threaded performance) can be somewhat better.  I haven't yet seen 
any benchmarks on larger disk arrays yet, eg 6+ disks, so no idea how it 
scales up.  Basically what I have seen seems competitive


I don't have such hardware spare to benchmark, but I would be interested 
to hear from someone who benchmarks your RAID1+linear+XFS suggestion, 
especially if they have compared a cutting edge btrfs kernel on the same 
array?


One of the snags of md RAID1 vs RAID6 is the lack of checksumming in the 
event of bad blocks.  (I'm not sure what actually happens when md 
scrubbing finds a bad sector with raid1..?).  For low performance 
requirements I have become paranoid and been using RAID6 vs RAID10, 
filesystems with sector checksums seem attractive...


Regards

Ed W

Re: [Dovecot] Authentication mechanism and Password scheme

2012-04-10 Thread Ed W


On 10/04/2012 08:11, Timo Sirainen wrote:

On 10.4.2012, at 5.37, Костырев Александр Алексеевич wrote:


Good day!
I'm just trying to figure out that my understanding of subject is correct.

So, if I want to store passwords in my database encrypted with SSHA512 scheme,
my only choice for Authentication mechanism is plaintext?

Yeah, that's correct.



Does dovecot 2.0 also support SCRAM-SHA?

I only mention because it's come up on my radar recently and as I 
understand it, it solves the issue of either having

- plain text db of passwords, encrypted login
- encrypted db of passwords, plaintext login

With SCRAM you have both sides encrypted.

(Clearly it's also desirable that the hash algorithm is well chosen to 
be resistant to bruteforce, so some might argue that bcrypt/scrypt is 
even more desirable since there is not yet a GPU implementation - 
However, at least SHA is a decent stab at things)


Can you confirm my understanding is correct?

Next question is whether any current mail client supports SCRAM..?

Regards

Ed W

Re: [Dovecot] dsync is SLOW compared to rsync

2012-03-25 Thread Ed W


On 24/03/2012 13:21, Maarten Bezemer wrote:


On Fri, 23 Mar 2012, Jeff Gustafson wrote:


That didn't seem to make much of a difference. On a 3.1GB backup it
shaved off 5 seconds. dsync's time was over 6 minutes with or without
the mail_fsync=never. rsync copied the same 3.1GB mailbox in 15 seconds.
It seems to me that dsync *should* be able to be just as fast, 
but it

currently is spending way too much time doing something. What is it?


Syncing 3.1GB in 15 seconds would require a speed of more than 200MB 
per second. Depending on the harddisks used, that would be quite a 
challenge.


rsync is only going to transfer files it believes has changed, so the 
transfer bandwidth will likely be lower


If you use rsync to only transfer the files that changed (based on 
file modification time) you may or may not miss files that have 
changed but still have the same time stamp. I assume you didn't use 
the --checksum parameter to rsync, right?


Dovecot is not very resiliant to files changing under it, but without 
the filename changing.  I have no idea if it's supposed to work at all, 
but you might at least expect to see problems if you start doing this?



dsync does so much more than simply copy some files...


Quite probably, but I don't think your expose above illustrates this?

Regards

Ed W

Re: [Dovecot] delivering with maildrop

2012-03-25 Thread Ed W


On 23/03/2012 14:02, Stan Hoeppner wrote:

On 3/23/2012 6:41 AM, Radim Kolar wrote:

Can somebody provide maildrop syntax for using deliver-lda as final
delivery program during sorting mail in user mailfilter?

i mean replacement  for to statement

if ( /^(To|Cc):.*dovecot@dovecot.org/:h )
{
  to $MAIL/.dovecot/
}

Dovecot's local delivery agent uses the Sieve language:
http://wiki.dovecot.org/LDA/Sieve

The syntax is quite different than maildrop or procmail.



I think that's why he asked the question?

I presume he wants to filter first with maildir, then actually deliver 
using the dovecot delivery agent?



In answer to the OP: read the maildropex man pages, but you have several 
options, eg:


to | someprogram
or:
xfilter someprogram
`someprogram`

However, almost certainly I think you want the top option?

Good luck

Ed W

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

2012-03-16 Thread Ed W


On 14/03/2012 10:58, Charles Marcus wrote:

On 2012-03-13 6:29 PM, Terry Carmen te...@cnysupport.com wrote:
I'm going to hope everything is OK for a while, since my goal is to 
retire

all the old Exchange servers and move all the users to dovecot/maildir
within the next couple of months.

However it's always nice to know there are options. 8-)


I'm currently looking at rolling out SOGo as part of a major reworking 
of their current infrastructure (will also include converting their 
old Courier-IMAP to dovecot 2.1.x among other things)...


SOGo, as far as I can tell, is the best truly free and open source 
'exchange clone' available that works extremely well with 
Thunderbird+Lightning (which is what my Client uses currently, but 
they are very dissatisfied with using Google Calendar for Shared 
calendars), Outlook and Apple Apps, as well as Android, Blackberry and 
Apple mobile devices - and their upcoming v2 (in beta now) will not 
only provide native Outlook support (no plugin needed), it will also 
(optionally) provide a Samba4 Active Directory server in my main 
Client's office - all with absolutely no licenses required. Commercial 
support is available from Inverse, the company created by the 
developers to provide said support services.


I also learned something very interesting yesterday concerning SOGo 
and dovecot during a sales call with a SOGo rep, but I'll wait and see 
if Timo cares to chime in on this one... ;)




If the answer is that he will write a Z-Push/Activesync module for SOGo 
then I'm all ears!  I have been watching SOGo for some time and the main 
thing I would miss is that every phone I have ever owned has largely 
limited/broken Funambol based sync and annoyingly working Activesync 
capability (I own a stream of Nokias...).  It seems that although I 
don't like it, I need activesync support if I want my contacts/calendar 
on my phone... (I think I can do caldav on some of them, but not cardav 
on my N9)


Apart from that it's a very neat system!

Ed W

Re: [Dovecot] Just in time AV scanning

2012-03-16 Thread Ed W


On 15/03/2012 10:33, Timo Sirainen wrote:

On Wed, 2012-03-14 at 16:51 -0700, Kelsey Cummings wrote:

I'm curious if anyone has any plugins for AV integration directly into
dovecot.

Our old pop servers have been scanning messges as they're moved from
new-cur in the inbox and,  at least where user's aren't poping every
few seconds, there is occasionally enough time between scanning through
the MXs to message retreval to snag a few more virues with updated
definitions before they reach customers.

Anyone doing anything similar?

http://dovecot.org/patches/2.1/mail-filter.tar.gz allows you to run a
script that modifies a mail while it's being read. You could make it run
a virus check, and if that happens you could change the virus MIME part
to be full of spaces (better not to change message size, line count or
MIME structure).




Couple of other ideas:

1) Could use one of the (buggy and variously unsupported) on access 
virus scanners.  I think Dazuko is now abandoned, but this is a new one 
mentioned via the Clamav site:

http://www.fsl.cs.sunysb.edu/docs/avfs-security04/index.html

2) Extremely racey, but if you were on maildir you could use some kind 
of pre-login scripting to kick off a scan on login.  Touch some lock 
file so that you can tell when last scanned and only scan if the 
definitions have been updated since you last scanned?


3) There are some POP proxies which offer inline virus scanning.  Could 
place one in front of your mail server.  Presumably this will expose you 
to all the bugs in that proxy...



Good luck

Ed W

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

2012-03-16 Thread Ed W


On 16/03/2012 15:45, Charles Marcus wrote:

On 2012-03-16 11:22 AM, Ed W li...@wildgooses.com wrote:

If the answer is that he will write a Z-Push/Activesync module for SOGo
then I'm all ears!  I have been watching SOGo for some time and the main
thing I would miss is that every phone I have ever owned has largely
limited/broken Funambol based sync and annoyingly working Activesync
capability (I own a stream of Nokias...).  It seems that although I
don't like it, I need activesync support if I want my contacts/calendar
on my phone... (I think I can do caldav on some of them, but not cardav
on my N9)


While I agree it would be nice, why not just switch to a supported 
phone and be done with it? ;)


When we roll out SOGo, we'll only be supporting the officially 
supported mobile clients (android, iphone/ipad, blackberry and windows 
mobile)...




That implies you will be using cardav/caldav on those phones?  I thought 
Android support was quite weak for those?


I definitely don't like the idea of supporting activesync, but it seems 
like the only widely supported solution to pushing calendar and contacts 
updates to clients?  Caldav gets you part of the way there, but cardav 
seems badly supported and there is no push support with either...


Out of curiousity, what kind of performance are you getting out of the 
web interface and any tricks you used to improve perceived 
performance? My quick testing gave something circa 150-200ms response 
times from SOGo (forget exactly now) and as a result it was perceivable 
and just very slightly laggy (versus a desktop mail program!!).  I get 
slightly better perceived performance from Roundcube (which also seems 
more amenable to building extension plugins)


Seems a bit of a surprise that a compiled language delivers results 
slightly less quickly than PHP... Did you find any magic knobs to twist 
to get performance up there with gmail?


Cheers

Ed W

Re: [Dovecot] Lock down Shared Mail Accounts?

2012-03-07 Thread Ed W



I want to give multiple people shared access to some actual accounts 
with all of the special use folders, with the following requirements:


I have done this (unsatisfactorarily) by making it a normal mail account 
with normal login credentials.  Add it like any other mail account.  It 
then satisfies all your requirements, although: behind a nat, on 
thunderbird and with condstore, I sometimes see read/unread get out of 
sync... Believed to be a thunderbird bug, but unsure. Easy to resync



5. No one other than a designated user or users (Master User(s)? Users 
in a specified Group?) can delete any messages in this account, in any 
of the folders.


Have them delivered with only read permissions on the physical files? 
(Bet that doesn't work very well in practice or other than maildir...)


Interested to hear proper answers...

Ed W

Re: [Dovecot] Post-login scripting - Trash cleanup

2012-02-28 Thread Ed W


On 28/02/2012 18:11, l...@airstreamcomm.net wrote:

We are considering using the post-login scripting to clear trash older
than 90 days from user accounts.  has anyone done this, and if so did this
cause logins to slow down too much waiting for the trash to purge?  One
idea was to execute the trash purge script once a day by tracking their
logins and seeing that it has already ran that day.  Another idea was to
call the trash purge script in the background and continue without
acknowledging that it has finished to keep logins speedy.


I think you can also use doveadm to achieve this?  So you could schedule 
something for all accounts at some out of hours period - should speed up 
backups also?


Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

2012-02-27 Thread Ed W


On 27/02/2012 08:34, Timo Sirainen wrote:

On Thu, 2012-02-23 at 01:41 +0200, Timo Sirainen wrote:


What do you need the statistics for? I could make imap_client and
pop3_client support some virtual methods, like user.destroy() initially,
which would be enough for your use. I guess I could add that for v2.2.

http://dovecot.org/patches/2.2/imap-logout-plugin.c




Thanks - can I assume that a pop-logout would be basically the same?

Also, how might I access the bytes in/out statistics from that context?

Thanks

Ed W

Re: [Dovecot] remove messages once downloaded

2012-02-26 Thread Ed W


On 25/02/2012 00:39, Timo Sirainen wrote:

On 24.2.2012, at 19.44, julio...@fisica.uh.cu wrote:


I need some help with the dovecot configuration. I want to remove
downloaded messages from Mail Server once the messages have been
successfully downloaded by pop3-clients, even when the clients have been
configured to save copy of messages in the Server.

Not possible.



If you were thinking about longer term TODOs then I have a similar 
problem (just adding a me too...)


In my industry, competing solutions offer a kind of server side been 
downloaded notification when customers have actually downloaded (ie 
read via POP) the message.  The customers are all on the far side of 
expensive satellite links, so this serves as an inexpensive proxy for 
message read notifications.


Is it feasible to implement both of these solutions using the current 
plugin architecture?


I think our competition implement such features because they are 
Exchange based and I believe you can write server side hooks in various 
scripting languages quite easily (I personally don't like the idea, but 
someone obviously did it once and it rolled from there...) - this 
obviously harking to the is it feasible to imagine some higher level 
hook solution for simpler plugin creation suggestion from a few days ago?


All these do something when it's accessed or do something when it's 
deleted problems all feel kind of related to me (ie we need some hook 
which runs on a per message basis). Perhaps someone smarter than me can 
think of a better way to unify them?


Cheers

Ed W

Re: [Dovecot] remove messages once downloaded

2012-02-26 Thread Ed W


On 26/02/2012 12:31, Timo Sirainen wrote:

On 26.2.2012, at 13.52, Ed W wrote:


On 25/02/2012 00:39, Timo Sirainen wrote:

On 24.2.2012, at 19.44, julio...@fisica.uh.cu wrote:


I need some help with the dovecot configuration. I want to remove
downloaded messages from Mail Server once the messages have been
successfully downloaded by pop3-clients, even when the clients have been
configured to save copy of messages in the Server.

Not possible.


If you were thinking about longer term TODOs then I have a similar problem (just adding a 
me too...)

In my industry, competing solutions offer a kind of server side been downloaded notification when 
customers have actually downloaded (ie read via POP) the message.  The customers are all on the 
far side of expensive satellite links, so this serves as an inexpensive proxy for message read 
notifications.

What does the notification do?


Sends another email... (you know like the annoying message read 
indicators that lots of mail readers support)...


(Several of our competitors have implemented these solutions very badly 
and we get mail loops and other nasties...)




I think our competition implement such features because they are Exchange based and I 
believe you can write server side hooks in various scripting languages quite easily (I 
personally don't like the idea, but someone obviously did it once and it rolled from 
there...) - this obviously harking to the is it feasible to imagine some higher 
level hook solution for simpler plugin creation suggestion from a few days ago?

All these do something when it's accessed or do something when it's deleted 
problems all feel kind of related to me (ie we need some hook which runs on a per message basis). 
Perhaps someone smarter than me can think of a better way to unify them?

Dovecot has a notify plugin that makes things like this pretty easy to 
implement, but it still needs C coding.


Thanks - it's off my radar for a while due to other pressures, but the 
hint is appreciated and I will look into it in the future - many thanks!


Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

2012-02-23 Thread Ed W


On 22/02/2012 23:56, Ed W wrote:
I think it has potential though.  I think a lot of the current plugins 
on the website could easily be rewritten, likely without performance 
concerns, using a scripting based plugin system.  I could see that 
some other big picture pieces could potentially benefit also


One interesting test case for such a scripting hooks solution might be 
login restrictions.  There seem to be regular requests for the ability 
to setup arbitrarily complicated restrictions on users per IP, attempts 
per second, etc (and my logging interest is kind of related also).


Not trying to bump the item up any todo lists, just trying to chuck in 
some concrete ideas for actually testing a specific implementation...


I guess a substantially more performance orientated area that seems to 
get some interest would be various spam, expunge, delete ideas and the 
hooks needed for those.  These seem much more tricky to implement a 
scripting hook and still stay performant.  Again just ideas for real 
things people might want to do?


Cheers

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend


On 22/02/2012 08:25, Jan-Frode Myklebust wrote:

On Tue, Feb 21, 2012 at 02:33:24PM +, Ed W wrote:

I think the original question was still sensible.  In your case it
seems like the ping times are identical between:
 webmail -  imap-proxy
 webmail -  imap server

I think your results show that a proxy has little (or negative)
benefit in this situation, but it seems feasible that a proxy could
eliminate several RTT trips in the event that the proxy is closer
than the imap server?  This might happen if say the imap server is
in a different datacenter (webmail on an office server machine?)

The webmail/imapproxy were actually running in a different datacenter to
the dovecot director/backend servers, but only about 20KM away.

Ping tests:

webmail-director:

rtt min/avg/max/mdev = 0.933/1.061/2.034/0.183 ms

director-backend:

rtt min/avg/max/mdev = 0.104/0.108/0.127/0.005 ms

webmail-localhost:

rtt min/avg/max/mdev = 0.020/0.062/1.866/0.257 ms


   -jf


Hmm, not sure I understand the original numbers then?

It seems intuitive that the proxy installed locally could save you 2x 
RTT increment, which is about 0.8ms in your case.  So I might expect the 
proxy to reduce rendering times by around 1.6ms simply because it 
reduces the number of round trips to login?  Kind of curious why that's 
not achieved..?


Cheers

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend


On 21/02/2012 20:36, Timo Sirainen wrote:

On 21.2.2012, at 16.33, Ed W wrote:


I'm also pleased to see that there is little negative cost in using a proxy... I recently added 
imap-proxy to our webmail setup because I wanted to log last login + logout times.  I 
haven't quite figured out how to best log logout time (Timo, any chance of a post 
logout script? Or perhaps it's possible with the current login scripting?).

You could of course grep the logs, but other than that you'd need to write a 
Dovecot plugin. Luckily it's really simple to write a plugin. Basically:

void postlogout_init(struct module *module) { }
void postlogout_deinit(void) {
   system(/usr/local/bin/dovecot-postlogout.sh);
}

Add a few missing #includes and compile and enable for imap/pop3 and that 
should be it.



Thanks - that's really obvious and quite interesting.  I guess a simple 
log plugin makes sense.


Quick followup question - the logout log file currently logs a bunch of 
statistics such as mails read/deleted, bytes sent/received.  How might I 
access these from the _deinit context as above?  Apologies if this is a 
RTFM question?


Finally, do you see it feasible to offer a scriptable plugin 
interface, eg perhaps using some high performance scripting language 
such as lua?  Such a plugin might itself be simply a standard plugin..?  
The motivation being to offer the ability to create plugins to those who 
are nervous of using a compiler, and of course to reduce the ability of 
a badly written plugin to kill dovecot?


Cheers

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend


On 22/02/2012 19:49, Timo Sirainen wrote:

On 22.2.2012, at 11.38, Ed W wrote:


void postlogout_init(struct module *module) { }
void postlogout_deinit(void) {
   system(/usr/local/bin/dovecot-postlogout.sh);
}

Add a few missing #includes and compile and enable for imap/pop3 and that 
should be it.

Thanks - that's really obvious and quite interesting.  I guess a simple log 
plugin makes sense.

Quick followup question - the logout log file currently logs a bunch of 
statistics such as mails read/deleted, bytes sent/received.  How might I access 
these from the _deinit context as above?  Apologies if this is a RTFM question?

You'd have to build separate plugins for POP3 and IMAP, and even then it 
becomes tricky since there's no simple hook for catching when client gets 
destroyed.


Do you think you could keep something similar on your low priority 
backlog?  Clearly parsing log files or hacking the code is possible, but 
I think the interest in the login scripting shows there is general 
interest, and having a full log of logon/logoff/bytes is clearly 
interesting to more minority users?



Finally, do you see it feasible to offer a scriptable plugin interface, eg 
perhaps using some high performance scripting language such as lua?  Such a plugin might 
itself be simply a standard plugin..?  The motivation being to offer the ability to 
create plugins to those who are nervous of using a compiler, and of course to reduce the 
ability of a badly written plugin to kill dovecot?

I've been thinking about adding a scripting language plugin to Dovecot. Perhaps 
even using one of the existing generators that are supposed to make this easy 
for multiple languages, such as SWIG. But this is pretty low priority 
currently..


I think SWIG is for wrapping dovecot's api into the scripting language?  
(ie you could call dovecot methods from say perl/python/etc). What I had 
in mind was the reverse, ie embed LUA inside dovecot.  Whenever dovecot 
normally calls a plugin method it will also run any [lua] scripts.


I'm sure you know how to use google, but just so we are on the same 
page, top hit (below) from google shows how straight forward this is 
(lua has been built to be extremely fast and easy to embed, ie it's not 
an arbitrary choice)

http://heavycoder.com/tutorials/lua_embed.php

Cheers

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend


On 22/02/2012 23:41, Timo Sirainen wrote:

I've heard LUA being a commonly used embedded language, but I'd prefer
to instead support several very widely used languages, such as
Perl/Python.


I'm a perl/ruby fan myself, but I would still recommend a good look at 
lua (or python) simply because they seem to be performant, easy to use, 
and on the surface seem to have had some thought about making them 
embeddable.


My new favourite editor Sublime Text 2 has python as it's scripting 
language.  Lua has been used for some big name games amongst other things.


Perl has some memory management issues if you leave it long running, 
also writing XS code looks ok on the surface, but is an exercise in hair 
pulling in practice  Ruby is a beautiful language, but unsure of how 
easy to embed and speed + memory management is an unknown (for high 
performance applications)


I think it has potential though.  I think a lot of the current plugins 
on the website could easily be rewritten, likely without performance 
concerns, using a scripting based plugin system.  I could see that some 
other big picture pieces could potentially benefit also


Thanks for considering it

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

2012-02-21 Thread Ed W


On 13/02/2012 19:43, Jan-Frode Myklebust wrote:

On Mon, Feb 13, 2012 at 11:08:48AM -0800, Mark Moseley wrote:

Out of curiosity, are you running dovecot locally on those webmail
servers as well, or is it talking to remote dovecot servers?

The webmail servers are talking with dovecot director servers which in
turn are talking with the backend dovecot servers. Each service running
on different servers.

Webmail-servers -  director-servers -  backend-servers



I think the original question was still sensible.  In your case it seems 
like the ping times are identical between:

webmail - imap-proxy
webmail - imap server

I think your results show that a proxy has little (or negative) benefit 
in this situation, but it seems feasible that a proxy could eliminate 
several RTT trips in the event that the proxy is closer than the imap 
server?  This might happen if say the imap server is in a different 
datacenter (webmail on an office server machine?)


I'm also pleased to see that there is little negative cost in using a 
proxy... I recently added imap-proxy to our webmail setup because I 
wanted to log last login + logout times.  I haven't quite figured out 
how to best log logout time (Timo, any chance of a post logout script? 
Or perhaps it's possible with the current login scripting?).  However, 
using imap-proxy has the benefit of clustering logins a little and 
this makes log files a little easier to understand in the face of users 
with desktop mail clients plus webmail users.  Possibly this idea useful 
to someone else...


Thanks for measuring this!

Ed W

Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?

2012-01-26 Thread Ed W


Hi

Sounds very cool. I already have dovecot set up as a proxy, working, 
and it should allow me to forcefully disconnect users and lock them 
out while they are being migrated and then once they are done they'll 
be served locally rather than proxied. My main problem is that most 
connections are simply coming directly to the old server, using the 
deprecated hostname. I need all clients to use the right hostnames, or 
clog up this new server with redirectors and proxies for all the junk 
done on the old server.. bummer.


Why not put the old server IP to redirect to the new machine, then give 
the old machine some new temp IP in order to proxy back to it?  That way 
you can do the proxying on the dovecot machine, which as you already 
established is working ok?


Good luck

Ed W

Re: [Dovecot] auth-worker temporary failures causing lmtp 500 rejection

2012-01-26 Thread Ed W


On 26/01/2012 14:37, Mark Zealey wrote:

I've tried reproducing by having long running auth queries in the sql and 
KILLing them on the server, restarting the mysql service, and setting max auth 
workers to 1 and running 2 sessions at the same time (with long-running auth 
queries), but to no effect. There must be something else going on here; I saw 
it in particular when exim on our frontend servers had queued a large number of 
messages and suddenly released them all at once hence the auth-worker 
hypothesis although the log messages do not support this. I'll try to see if I 
can trigger this manually although we have been doing some massively parallel 
testing previously and not seen this.



Could it be a *timeout* rather than lack of worker processes?  Theory 
would be that disk starvation causes other processes to take a long time 
to respond, hence the worker is *alive*, but doesn't return a response 
quickly enough, which in turn causes the unknown user message?


You could try a different disk io scheduler, or ionice to control the 
effect of these big bursts of disk activity on other processes?


(Most MTA programs such as postfix and qmail do a lot of fsyncs - this 
will cause a lot of IO activity and could easily starve other processes 
on the same box?)



Good luck

Ed W

Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?

2012-01-26 Thread Ed W


Hi

Yeap, taht's what I'm doing to do, except that I would have to proxy 
more than just IMAP and POP - it's a one-does-it-all kind of machine 
accepting mail delivered from the outside, relaying outgoing mail, 
does webmail, does all this things very poorly... I have the choice of 
forcing all users to change to the new, dedicated servers doing these 
things, or reimplementing / porxying all of this on my new dovecot 
server which I so desperately want to keep neat and tidy...


In that case I would suggest perhaps that the IP is taken over by a 
dedicated firewall box (running the OS of your choice).  The firewall 
could then be used to port forward the services to the individual 
machines responsible for each service.  This would give you the benefit 
that you could easily move other services off/around


We are clearly off topic to dovecot...

Plenty of good firewall options.  If you want small, compact and low 
power, then you can pickup a bunch off intel compatible boards around 
the low couple hundred £s mark fairly easily.  Run your favourite distro 
and firewall on them.  If you hadn't seen them before, I quite like 
Lanner for appliances, eg:

http://www.lannerinc.com/x86_Network_Appliances/x86_Desktop_Appliances

For example if you added a small appliance running linux which runs that 
IP, then you could add intrusion detection, bounce the web traffic to 
the windows box (or even just certain URLs, other URLs could go to some 
hypothetical linux box, etc), port forwarding the mail to the new 
dovecot box, etc, etc.  Incremental price would be surprisingly low, but 
lots of extra flexibility?


Just a thought

Good luck

Ed W

[Dovecot] Password auth scheme question with mysql

Hi, I have a current auth database using mysql with a password column 
in plain text. The config has default_pass_scheme = PLAIN specified


In preparation for a more adaptable system I changed a password entry 
from asdf to {PLAIN}asdf, but now auth fails.  Works fine if I 
change it back to just asdf. (I don't believe it's a caching problem)


What might I be missing?  I was under the impression that the password 
column can include a {scheme} prefix to indicate the password scheme 
(presumably this also means a password cannot start with a {?).  Is 
this still true when using mysql and default_pass_scheme ?


Thanks for any hints?

Ed W

Re: [Dovecot] Password auth scheme question with mysql


On 24/01/2012 22:06, Ed W wrote:
Hi, I have a current auth database using mysql with a password 
column in plain text. The config has default_pass_scheme = PLAIN 
specified


In preparation for a more adaptable system I changed a password entry 
from asdf to {PLAIN}asdf, but now auth fails.  Works fine if I 
change it back to just asdf. (I don't believe it's a caching problem)


What might I be missing?  I was under the impression that the password 
column can include a {scheme} prefix to indicate the password scheme 
(presumably this also means a password cannot start with a {?).  Is 
this still true when using mysql and default_pass_scheme ?


Hmm, so I try:

# doveadm pw -p asdf -s sha256
{SHA256}8OTC92xYkW7CWPJGhRvqCR0U1CR6L8PhhpRGGxgW4Ts=

I enter this hash into my database column, then enabling debug logging I 
see this in the logs:


Jan 24 22:40:44 mail1 dovecot: auth: Debug: 
cache(d...@mailasail.com,1.2.24.129): SHA256({PLAIN}asdf) != 
'8OTC92xYkW7CWPJGhRvqCR0U1CR6L8PhhpRGGxgW4Ts='
Jan 24 22:40:44 mail1 dovecot: auth-worker: Debug: 
sql(d...@blah.com,1.2.24.129): query: SELECT NULLIF(mail_host, 
'1.2.24.129') as proxy, NULLIF(mail_host, '1.2.24.129') as host, email 
as user, password, password as pass, home userdb_home, concat(home, '/', 
maildir) as userdb_mail, 200 as userdb_uid, 200 as userdb_gid FROM users 
WHERE email = 
if('blah.com''','d...@blah.com','d...@blah.com@mailasail.com') and 
flag_active=1
Jan 24 22:40:44 mail1 dovecot: auth-worker: 
sql(d...@blah.com,1.2.24.129): Password mismatch (given password: 
{PLAIN}asdf)
Jan 24 22:40:44 mail1 dovecot: auth-worker: Error: 
md5_verify(d...@mailasail.com): Not a valid MD5-CRYPT or PLAIN-MD5 password
Jan 24 22:40:44 mail1 dovecot: auth-worker: Error: 
ssha256_verify(d...@mailasail.com): SSHA256 password too short
Jan 24 22:40:44 mail1 dovecot: auth-worker: Error: 
ssha512_verify(d...@mailasail.com): SSHA512 password too short
Jan 24 22:40:44 mail1 dovecot: auth-worker: Warning: Invalid OTP data in 
passdb
Jan 24 22:40:44 mail1 dovecot: auth-worker: Warning: Invalid OTP data in 
passdb
Jan 24 22:40:44 mail1 dovecot: auth-worker: Debug: 
sql(d...@blah.com,1.2.24.129): SHA256({PLAIN}asdf) != 
'8OTC92xYkW7CWPJGhRvqCR0U1CR6L8PhhpRGGxgW4Ts='



Forgot to say. this is with dovecot 2.0.17

Thanks for any pointers

Ed W

Re: [Dovecot] Password auth scheme question with mysql


On 24/01/2012 22:51, Ed W wrote:

Hmm, so I try:

# doveadm pw -p asdf -s sha256
{SHA256}8OTC92xYkW7CWPJGhRvqCR0U1CR6L8PhhpRGGxgW4Ts=

I enter this hash into my database column, then enabling debug logging 
I see this in the logs:



..
Jan 24 22:40:44 mail1 dovecot: auth-worker: Debug: 
sql(d...@blah.com,1.2.24.129): SHA256({PLAIN}asdf) != 
'8OTC92xYkW7CWPJGhRvqCR0U1CR6L8PhhpRGGxgW4Ts='


Gah.  Ok, so I discovered the doveadm auth command:


# doveadm auth -x service=pop3 demo asdf
passdb: demo auth succeeded
extra fields:
  user=d...@blah.com
  proxy
  host=1.2.24.129
  pass={SHA256}8OTC92xYkW7CWPJGhRvqCR0U1CR6L8PhhpRGGxgW4Ts=



So why do I get an auth failed and the log files I showed in my last 
email when I use telnet localhost 110 and then the commands:


user demo
pass asdf


Help please...?

Ed W

Re: [Dovecot] Password auth scheme question with mysql