Re: [Dovecot] Update problem from 1.2 = 2.0.19 and recommended imap storage

2012-03-17 Thread Gerhard Wiesinger

On Fri, 16 Mar 2012, Timo Sirainen wrote:


On 16.3.2012, at 8.02, Gerhard Wiesinger wrote:

Calling imap still fails as non root:
imap
/usr/bin/ld: cannot open output file /usr/local/bin/.libs/2612-lt-imap: 
Permission denied
collect2: ld returned 1 exit statusn


Huh? That looks like imap is running ld to link something. It shouldn't be 
doing that.


After starting it once as root the following files are created and it 
works also as non root:

ls -l /usr/local/bin/.libs/
total 1160
-rwxr-xr-x  1 root root 235848 Aug 25  2010 lt-doveconf
-rwxr-xr-x  1 root root 938454 Mar 16 07:03 lt-imap

Before only one of these files was generated (I think lt-doveconf).

Strange. Any ideas?

Ciao,
Gerhard

--
http://www.wiesinger.com/


[Dovecot] importing plain mboxes to dovecot maildirs

2012-03-17 Thread Radim Kolar
Is there way to import old plain mboxes via dsync? It complains about 
lack of index files:


ponto:(admin)~dsync mirror mbox:~/mail
dsync(admin): Error: Failed to sync mailbox sent-mail: Mailbox GUIDs are 
not permanent without index files
dsync(admin): Error: Failed to sync mailbox saved-messages: Mailbox 
GUIDs are not permanent without index files
dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox 
GUIDs are not permanent without index files




Re: [Dovecot] importing plain mboxes to dovecot maildirs

2012-03-17 Thread Timo Sirainen
On 17.3.2012, at 8.36, Radim Kolar wrote:

 Is there way to import old plain mboxes via dsync? It complains about lack of 
 index files:
 
 ponto:(admin)~dsync mirror mbox:~/mail
 dsync(admin): Error: Failed to sync mailbox sent-mail: Mailbox GUIDs are not 
 permanent without index files
 dsync(admin): Error: Failed to sync mailbox saved-messages: Mailbox GUIDs are 
 not permanent without index files
 dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs 
 are not permanent without index files

Well, you can work around if by letting it create indexes. Hm. Why exactly 
can't it create indexes? Do you have some setting disabling them?



Re: [Dovecot] POP3 Performance

2012-03-17 Thread Mark Alan
On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen t...@iki.fi wrote:
 On 16.3.2012, at 13.07, Mauricio López Riffo wrote:

 pop3_no_flag_changes=yes

Is it the same as pop3_no_flag_updates=yes ?


M.


Re: [Dovecot] POP3 Performance

2012-03-17 Thread Timo Sirainen
On 17.3.2012, at 16.14, Mark Alan wrote:

 On Fri, 16 Mar 2012 15:07:24 +0200, Timo Sirainen t...@iki.fi wrote:
 On 16.3.2012, at 13.07, Mauricio López Riffo wrote:
 
 pop3_no_flag_changes=yes
 
 Is it the same as pop3_no_flag_updates=yes ?

Yeah. I wrote it from my memory.



Re: [Dovecot] importing plain mboxes to dovecot maildirs

2012-03-17 Thread Radim Kolar



dsync(admin): Error: Failed to sync mailbox sent-mail-feb-2012: Mailbox GUIDs 
are not permanent without index files
Well, you can work around if by letting it create indexes. Hm. Why exactly 
can't it create indexes? Do you have some setting disabling them?
indexes never existed because these mboxes were never used by dovecot, 
its not conversion from one format to another, its import.


Maybe open bug to add feature dsync import which will not depend on 
existing indexes?


[Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Kaya Saman

Hi,

I am currently in the process of setting up an IMAP repository for round 
100 users


Currently the user authentication method is being handled via a Windows 
Domain Controller.


The host OS for Dovecot will either be FreeBSD or CentOS.


Would Dovecot be able to authenticate to either the DC directly or would 
we need to go through LDAP??



Additionally what would be the best method to store the **mail** 
information? - as in MySQL database or Maildir format; coinciding with 
this what is the best backup method in order to be able to do 'dump' 
backups or restore single emails??



Can anyone give me a hand with this?


Regards,


Kaya


Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Sven Hartge
Kaya Saman kayasa...@gmail.com wrote:

 I am currently in the process of setting up an IMAP repository for
 round 100 users

 Currently the user authentication method is being handled via a
 Windows Domain Controller.

 The host OS for Dovecot will either be FreeBSD or CentOS.

 Would Dovecot be able to authenticate to either the DC directly or
 would we need to go through LDAP??

Why not join the server to the domain and simply use PAM? 

Using ActiveDirectory through LDAP is a bit of a pain so I would avoid
this if I where you.

 Additionally what would be the best method to store the **mail**
 information? - as in MySQL database or Maildir format; coinciding with
 this what is the best backup method in order to be able to do 'dump'
 backups or restore single emails??

Storing mails inside SQL? Not supported by dovecot and not very wise,
IMHO. DBmail does this, but to be honest, I never heard any good
feedback from admins using that product. From what I have been told, you
need quite the beefy server to get a decent performance out of DBmail,
compared to the needs of a traditional setup like with dovecot or
courier-mail, but I digress.

To have a consistent backup, your mail storage should be able to
snapshot the volume the mail is stored on, so use LVM or an external
storage unit capable of snapshots.

Then backup the content of the snapshot using any program you like.
I use Bacula for long-term offsite storage and a local rsnapshot to keep
7 days worth of mail for a quick restore.

Whether you are able to restore single mails or the complete storage is
no property or feature of the mailbox format itself.

Some formats are simpler to handle, like Maildir++, where you just drop
the file containing a mail into a directory.

Some, like mbox or mdbox are a little bit more complex, but with the
correct doveadm command you are nevertheless able to restore single
mails.


Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.



[Dovecot] Problem managing mbox

2012-03-17 Thread PSTM

Hello,

I have a problem with dovecot. seems that do not erase mail that mail 
client request to be erased.

And I have this errors:

 Error: Next message unexpectedly corrupted in mbox file

Info:


dovecot-2.1.1-2.0.cf.fc16.i686



root  5979  0.0  0.1   3208  1260 ?Ss   20:18   0:00 
/usr/sbin/dovecot -F



dovenull  5985  0.0  0.2   7060  2280 ?S20:18   0:00 
dovecot/imap-login
vmail 5988  0.0  0.1   7888  1848 ?S20:18   0:00 
dovecot/imap


permissions on mail dir:

total 4
drwxr-xr-x 9 vmail mail 4096 ene 21 21:43 vmail



Any suggestion?

Regards,


--
--

http://www.0pc.eu/


Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Kaya Saman

On 03/17/2012 07:36 PM, Sven Hartge wrote:

Kaya Samankayasa...@gmail.com  wrote:


I am currently in the process of setting up an IMAP repository for
round 100 users
Currently the user authentication method is being handled via a
Windows Domain Controller.
The host OS for Dovecot will either be FreeBSD or CentOS.
Would Dovecot be able to authenticate to either the DC directly or
would we need to go through LDAP??

Why not join the server to the domain and simply use PAM?

Using ActiveDirectory through LDAP is a bit of a pain so I would avoid
this if I where you.


Danke Sven :-)

I don't actually have much AD/LDAP integration experience so I will try 
your method!



Additionally what would be the best method to store the **mail**
information? - as in MySQL database or Maildir format; coinciding with
this what is the best backup method in order to be able to do 'dump'
backups or restore single emails??

Storing mails inside SQL? Not supported by dovecot and not very wise,
IMHO. DBmail does this, but to be honest, I never heard any good
feedback from admins using that product. From what I have been told, you
need quite the beefy server to get a decent performance out of DBmail,
compared to the needs of a traditional setup like with dovecot or
courier-mail, but I digress.

To have a consistent backup, your mail storage should be able to
snapshot the volume the mail is stored on, so use LVM or an external
storage unit capable of snapshots.


Hmm. so FreeBSD coupled together with a ZFS repo for mail should 
take care of 'Snapshot' issues.




Then backup the content of the snapshot using any program you like.
I use Bacula for long-term offsite storage and a local rsnapshot to keep
7 days worth of mail for a quick restore.


To be honest I was considering rsync'ing the dir containing users 
mailboxes to either another storage pool or server.




Whether you are able to restore single mails or the complete storage is
no property or feature of the mailbox format itself.

Some formats are simpler to handle, like Maildir++, where you just drop
the file containing a mail into a directory.


You mention Maildir++... is this Maildir format or something new which I 
haven't heard about yet?




Some, like mbox or mdbox are a little bit more complex, but with the
correct doveadm command you are nevertheless able to restore single
mails.


Grüße,
Sven.



Regards,

Kaya


Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Sven Hartge
Kaya Saman kayasa...@gmail.com wrote:
 On 03/17/2012 07:36 PM, Sven Hartge wrote:
 Kaya Samankayasa...@gmail.com  wrote:

 I am currently in the process of setting up an IMAP repository for
 round 100 users  Currently the user authentication method is
 being handled via a Windows Domain Controller.  The host OS for
 Dovecot will either be FreeBSD or CentOS.  Would Dovecot be able to
 authenticate to either the DC directly or would we need to go
 through LDAP??

 Why not join the server to the domain and simply use PAM?

 Using ActiveDirectory through LDAP is a bit of a pain so I would
 avoid this if I where you.

 I don't actually have much AD/LDAP integration experience so I will
 try your method!

Question: do you need public or shared folders?

Using samba and winbindd to join a domain creates real users on your
server and as far as I know configuring shared folders with real users
is a bit of a pain, especially of you need shared flags (like Seen,
Replied, etc.) (Someone [Timo?] please correct me.)

 Additionally what would be the best method to store the **mail**
 information? - as in MySQL database or Maildir format; coinciding
 with this what is the best backup method in order to be able to do
 'dump' backups or restore single emails??

 Storing mails inside SQL? Not supported by dovecot and not very wise,
 IMHO. DBmail does this, but to be honest, I never heard any good
 feedback from admins using that product. From what I have been told, you
 need quite the beefy server to get a decent performance out of DBmail,
 compared to the needs of a traditional setup like with dovecot or
 courier-mail, but I digress.

 To have a consistent backup, your mail storage should be able to
 snapshot the volume the mail is stored on, so use LVM or an external
 storage unit capable of snapshots.

 Hmm. so FreeBSD coupled together with a ZFS repo for mail should 
 take care of 'Snapshot' issues.

Yes. Or using LVM on Linux.

 Then backup the content of the snapshot using any program you like.
 I use Bacula for long-term offsite storage and a local rsnapshot to
 keep 7 days worth of mail for a quick restore.

 To be honest I was considering rsync'ing the dir containing users
 mailboxes to either another storage pool or server.

No need to rsync, if you use ZFS. Just create a new snapshot and you are
done. Bet thing about ZFS: you get deduplication for free, so the needed
space to store the backups will not grow as fast.

But you still may want to store the mails offsite/offserver for desaster
recovery.

Either use doveadm backup for that purpose or use rsnapshot, again
gaining you deduplication on the target server.

 Whether you are able to restore single mails or the complete storage is
 no property or feature of the mailbox format itself.

 Some formats are simpler to handle, like Maildir++, where you just drop
 the file containing a mail into a directory.

 You mention Maildir++... is this Maildir format or something new which I 
 haven't heard about yet?

Maildir++ extends the original Maildir with things like Quota and ACLs
and was first implemented in Courier.
http://www.courier-mta.org/imap/README.maildirquota.html

All current MTAs and POP3/IMAP servers implement this variant.

Depending on the amount of mail a user collects inside a folder, Maildir
is not the best storage format. You may want to check into mdbox, if
your users are kind of mail hoarders (like some of my users are).

In my opinion, Maildir has outlived its usefullnes. It was fine when
users had 1,000 mails in some 10 folders, but today, users collect over
100,000 mails a year and Maildir is causing serious I/O trouble and the
need to heavily fine tune your storage and filesystems to cope with
those demands.

I cannot thank Timo enough for inventing mdbox, as this format breaks
this viciuos cycle and, as someone else said it ends the battle at the
I/O front forever.

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.



Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Kaya Saman

Thanks so much Sven for your indepth and complete responses!



snip

Question: do you need public or shared folders?


I don't need anything apart from an IMAP storage solution. I don't 
intend to tie in Dovecot with an MTA either as I will simply be using 
this for storage.


Long story but we don't have any control over our mail server which is 
handled by the parent company abroad and is on MS Exchange.


To use an IMAP storage solution is the only way to get rid of pesky MS 
.pst files which have been causing everyone grief and havoc.




Using samba and winbindd to join a domain creates real users on your
server and as far as I know configuring shared folders with real users
is a bit of a pain, especially of you need shared flags (like Seen,
Replied, etc.) (Someone [Timo?] please correct me.)


Actually we might have an LDAP server already taking care of the 
AD-UNIX integration. I don't know yet it's only my first week :-)





Additionally what would be the best method to store the **mail**
information? - as in MySQL database or Maildir format; coinciding
with this what is the best backup method in order to be able to do
'dump' backups or restore single emails??

Storing mails inside SQL? Not supported by dovecot and not very wise,
IMHO. DBmail does this, but to be honest, I never heard any good
feedback from admins using that product. From what I have been told, you
need quite the beefy server to get a decent performance out of DBmail,
compared to the needs of a traditional setup like with dovecot or
courier-mail, but I digress.

To have a consistent backup, your mail storage should be able to
snapshot the volume the mail is stored on, so use LVM or an external
storage unit capable of snapshots.

Hmm. so FreeBSD coupled together with a ZFS repo for mail should
take care of 'Snapshot' issues.

Yes. Or using LVM on Linux.


Yeah true but I specified ZFS as I'm a fan and also am quite 
comfortable with Solaris/*BSD too..






Then backup the content of the snapshot using any program you like.
I use Bacula for long-term offsite storage and a local rsnapshot to
keep 7 days worth of mail for a quick restore.

To be honest I was considering rsync'ing the dir containing users
mailboxes to either another storage pool or server.

No need to rsync, if you use ZFS. Just create a new snapshot and you are
done. Bet thing about ZFS: you get deduplication for free, so the needed
space to store the backups will not grow as fast.


Ok so that solves that! :-)


But you still may want to store the mails offsite/offserver for desaster
recovery.


They are currently being stored on the parent company mail server so 
this will be the/off-site/ disaster recovery system in a way :-P




Either use doveadm backup for that purpose or use rsnapshot, again
gaining you deduplication on the target server.


I will research this - thank you for that info :-)




Whether you are able to restore single mails or the complete storage is
no property or feature of the mailbox format itself.

Some formats are simpler to handle, like Maildir++, where you just drop
the file containing a mail into a directory.

You mention Maildir++... is this Maildir format or something new which I
haven't heard about yet?

Maildir++ extends the original Maildir with things like Quota and ACLs
and was first implemented in Courier.
http://www.courier-mta.org/imap/README.maildirquota.html

All current MTAs and POP3/IMAP servers implement this variant.

Depending on the amount of mail a user collects inside a folder, Maildir
is not the best storage format. You may want to check into mdbox, if
your users are kind of mail hoarders (like some of my users are).

In my opinion, Maildir has outlived its usefullnes. It was fine when
users had 1,000 mails in some 10 folders, but today, users collect over
100,000 mails a year and Maildir is causing serious I/O trouble and the
need to heavily fine tune your storage and filesystems to cope with
those demands.

I cannot thank Timo enough for inventing mdbox, as this format breaks
this viciuos cycle and, as someone else said it ends the battle at the
I/O front forever.


So mdbox is a 'new' mailbox standard? ie. one can replace Maildir format 
with this and use mdbox instead. {Note to self: time to browse!}


Since where I'm implementing this is mainly an MS based environment they 
are concerned about /flat/ files which MS seems to typically do 
(although never used MS before so I wouldn't know). So there is some 
concern over performance, efficiency and manageability.


However, if like you say mdbox is the way to go then I will put a strong 
case together!




Grüße,
Sven.



Regards,


Kaya


Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Spyros Tsiolis
Hi,

I am currently in the process of setting up an IMAP repository for round 100 
users

Currently the user authentication method is being handled via a Windows Domain 
Controller.

The host OS for Dovecot will either be FreeBSD or CentOS.


Would Dovecot be able to authenticate to either the DC directly or would we 
need to go through LDAP??


Additionally
 what would be the best method to store the **mail** information? - as 
in MySQL database or Maildir format; coinciding with this what is the 
best backup method in order to be able to do 'dump' backups or restore 
single emails??


Can anyone give me a hand with this?


Regards,


Kaya




Hi Kaya,

I can't force you to follow a specific path.
All I can do, is tell you my experience on this.

Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for
Webmail, OpenLDAP for LDAP (no windows software
there) and CentOS v5.5 32-bit onwards.
User base is about 30 users.
System uptime without a glitch reached at some point
(had to reboot the server for maintenance reasons) about
200 days. I am sure it would go beyond 365 days.

Hope this helps,

spyros




I merely function as a channel that filters 
music through the chaos of noise
- Vangelis


Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Kaya Saman

On 03/17/2012 09:51 PM, Spyros Tsiolis wrote:

Hi,

I am currently in the process of setting up an IMAP repository for round 100 
users

Currently the user authentication method is being handled via a Windows Domain 
Controller.

The host OS for Dovecot will either be FreeBSD or CentOS.


Would Dovecot be able to authenticate to either the DC directly or would we 
need to go through LDAP??


Additionally

  what would be the best method to store the **mail** information? - as
in MySQL database or Maildir format; coinciding with this what is the
best backup method in order to be able to do 'dump' backups or restore
single emails??


Can anyone give me a hand with this?


Regards,


Kaya




Hi Kaya,

I can't force you to follow a specific path.
All I can do, is tell you my experience on this.

Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for
Webmail, OpenLDAP for LDAP (no windows software
there) and CentOS v5.5 32-bit onwards.
User base is about 30 users.
System uptime without a glitch reached at some point
(had to reboot the server for maintenance reasons) about
200 days. I am sure it would go beyond 365 days.

Hope this helps,

spyros




I merely function as a channel that filters
music through the chaos of noise
- Vangelis


Thanks for that Spyros!


Regards,


Kaya


Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Sven Hartge
Kaya Saman kayasa...@gmail.com wrote:
 snip
 Question: do you need public or shared folders?

 I don't need anything apart from an IMAP storage solution. I don't
 intend to tie in Dovecot with an MTA either as I will simply be using
 this for storage.

 Long story but we don't have any control over our mail server which is
 handled by the parent company abroad and is on MS Exchange.

 To use an IMAP storage solution is the only way to get rid of pesky MS
 .pst files which have been causing everyone grief and havoc.

So, how do you plan to get the mails into this storage? offlineimap?
imapsync? mbsync? fetchmail?

 Hmm. so FreeBSD coupled together with a ZFS repo for mail should
 take care of 'Snapshot' issues.
 Yes. Or using LVM on Linux.

 Yeah true but I specified ZFS as I'm a fan and also am quite
 comfortable with Solaris/*BSD too..

If you know ZFS and are familiar with it, then, by all means, go for it.

 Depending on the amount of mail a user collects inside a folder,
 Maildir is not the best storage format. You may want to check into
 mdbox, if your users are kind of mail hoarders (like some of my
 users are).

 In my opinion, Maildir has outlived its usefullnes. It was fine when
 users had 1,000 mails in some 10 folders, but today, users collect
 over 100,000 mails a year and Maildir is causing serious I/O trouble
 and the need to heavily fine tune your storage and filesystems to
 cope with those demands.

 I cannot thank Timo enough for inventing mdbox, as this format breaks
 this viciuos cycle and, as someone else said it ends the battle at
 the I/O front forever.

 So mdbox is a 'new' mailbox standard? ie. one can replace Maildir
 format with this and use mdbox instead. {Note to self: time to
 browse!}

mdbox is a format invented by Timo for dovecot. But dovecot can use
nearly all common mailbox formats (except MH, but no one uses that one
today).

 Since where I'm implementing this is mainly an MS based environment
 they are concerned about /flat/ files which MS seems to typically
 do (although never used MS before so I wouldn't know). So there is
 some concern over performance, efficiency and manageability.

Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no
doubt about this.

But even with this crappy format, dovecot is able to deliver astounding
performance by use of separete index files which allow it to access the
storage in an efficient manner.

mbox has big problems with concurrent writes, the bigger the mbox is,
the more problems you get. This is mainly caused by the meta-data of a
message (meaning flags, status, etc.) which is stored inside the mbox
file itself. Flagging a message as read or replied causes the whole mbox
file to be rewritten.

mdbox solves this problem by a) storing all meta-data in the index and
b) by only ever appending to a mdbox storage file, c) never
truncating an existing mdbox storage file and d) using more than one
mdbox storage file. Max size and TTL are configurable.

But this also means deleted mails are still inside a mdbox storage file
and need to be finally removed by copying all remaining files into a new
file. This process has to be manually run during low traffic hours, for
example using a cronjob.

You can say, mdbox is like mbox on steroids. ;)

Flat files are not evil or bad or slow per se, but you have to use them
the right way.

 However, if like you say mdbox is the way to go then I will put a
 strong case together!

You may want to start with something familiar and convert later, which
is no problem with dovecot.

Grüße,
Sven

-- 
Sigmentation fault. Core dumped.



Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Kaya Saman

On 03/17/2012 10:28 PM, Sven Hartge wrote:

Kaya Samankayasa...@gmail.com  wrote:

snip

Question: do you need public or shared folders?

I don't need anything apart from an IMAP storage solution. I don't
intend to tie in Dovecot with an MTA either as I will simply be using
this for storage.
Long story but we don't have any control over our mail server which is
handled by the parent company abroad and is on MS Exchange.
To use an IMAP storage solution is the only way to get rid of pesky MS
.pst files which have been causing everyone grief and havoc.

So, how do you plan to get the mails into this storage? offlineimap?
imapsync? mbsync? fetchmail?


Since everything is blocked at the Exchange end, users will have to 
manually transfer for now through MS Outlook.


Currently that's what they're doing to their PST's




Hmm. so FreeBSD coupled together with a ZFS repo for mail should
take care of 'Snapshot' issues.

Yes. Or using LVM on Linux.

Yeah true but I specified ZFS as I'm a fan and also am quite
comfortable with Solaris/*BSD too..

If you know ZFS and are familiar with it, then, by all means, go for it.


:-)




Depending on the amount of mail a user collects inside a folder,
Maildir is not the best storage format. You may want to check into
mdbox, if your users are kind of mail hoarders (like some of my
users are).

In my opinion, Maildir has outlived its usefullnes. It was fine when
users had 1,000 mails in some 10 folders, but today, users collect
over 100,000 mails a year and Maildir is causing serious I/O trouble
and the need to heavily fine tune your storage and filesystems to
cope with those demands.

I cannot thank Timo enough for inventing mdbox, as this format breaks
this viciuos cycle and, as someone else said it ends the battle at
the I/O front forever.

So mdbox is a 'new' mailbox standard? ie. one can replace Maildir
format with this and use mdbox instead. {Note to self: time to
browse!}

mdbox is a format invented by Timo for dovecot. But dovecot can use
nearly all common mailbox formats (except MH, but no one uses that one
today).


Ok so if you claim that mdbox is the 'best' mailbox storage solution 
then I'll look at implementing this.





Since where I'm implementing this is mainly an MS based environment
they are concerned about /flat/ files which MS seems to typically
do (although never used MS before so I wouldn't know). So there is
some concern over performance, efficiency and manageability.

Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no
doubt about this.

But even with this crappy format, dovecot is able to deliver astounding
performance by use of separete index files which allow it to access the
storage in an efficient manner.

mbox has big problems with concurrent writes, the bigger the mbox is,
the more problems you get. This is mainly caused by the meta-data of a
message (meaning flags, status, etc.) which is stored inside the mbox
file itself. Flagging a message as read or replied causes the whole mbox
file to be rewritten.

mdbox solves this problem by a) storing all meta-data in the index and
b) by only ever appending to a mdbox storage file, c) never
truncating an existing mdbox storage file and d) using more than one
mdbox storage file. Max size and TTL are configurable.

But this also means deleted mails are still inside a mdbox storage file
and need to be finally removed by copying all remaining files into a new
file. This process has to be manually run during low traffic hours, for
example using a cronjob.

You can say, mdbox is like mbox on steroids. ;)

Flat files are not evil or bad or slow per se, but you have to use them
the right way.


Thanks a lot for that info. I will research more into this but I maybe 
overridden at some point :-(


Need to make a strong case!




However, if like you say mdbox is the way to go then I will put a
strong case together!

You may want to start with something familiar and convert later, which
is no problem with dovecot.


Maildir is what I'm familiar with currently and mbox format - though 
only use mbox as an unfortunate side product of /system mail/ accounts.


Works well with Alpine client though!



Grüße,
Sven



Regards,

Kaya


Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Sven Hartge
Kaya Saman kayasa...@gmail.com wrote:

 Flat files are not evil or bad or slow per se, but you have to use
 them the right way.

 Thanks a lot for that info. I will research more into this but I maybe
 overridden at some point :-(

 Need to make a strong case!

Hmm.

Just because Microsofts way of usage of flat file database sucks does
not mean any usage of flat files is bad or evil or slow, if done right.

Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox 

But as I wrote before, it is quite easy to convert from one format to
the other: http://wiki2.dovecot.org/Migration/MailFormat

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.



Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Kaya Saman

On 03/18/2012 12:04 AM, Sven Hartge wrote:

Kaya Samankayasa...@gmail.com  wrote:


Flat files are not evil or bad or slow per se, but you have to use
them the right way.

Thanks a lot for that info. I will research more into this but I maybe
overridden at some point :-(
Need to make a strong case!

Hmm.

Just because Microsofts way of usage of flat file database sucks does
not mean any usage of flat files is bad or evil or slow, if done right.


Coming from a UNIX background I deal quite a lot with this kind of stuff 
so there's not problem for me. However, where I'm trying to deploy this 
system is a primarily MS based enterprise meaning that as the only UNIX 
engineer onsite and the newest addition to the team I have to convince 
people of working with UNIX technologies or somehow increase UNIX awareness.


As a bi-product I know nothing about MS tech. only what it told to me by 
my colleagues :-)




Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox


I checked that out after your last email... I started Google'ing a 
little. :-)


Looks like it would be a good solution!



But as I wrote before, it is quite easy to convert from one format to
the other: http://wiki2.dovecot.org/Migration/MailFormat


Once we get setup this may come in quite handy! Not sure what's going on 
currently as everyone above me is still quite set in using an SQL DB as 
a mail storage system???


To be honest, I run Zimbra @home for my OpenSource work and really enjoy 
it; in conjunction with Dovecot on FreeBSD which I run imapsync to 
backup **all** emails to.


It works really well.. :-)

I have messed around with Postfix, Dovecot and Horde3 in the past which 
also was really nice.




Grüße,
Sven.


Regards,


Kaya


Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Sven Hartge
Kaya Saman kayasa...@gmail.com wrote:

 Once we get setup this may come in quite handy! Not sure what's going
 on currently as everyone above me is still quite set in using an SQL
 DB as a mail storage system???

RDBMS where not designed for such a task.  Using a relational database
as a storage method for big chunks of data is very unwise, in my
opinion. It degrades them to just being some sort of filing cabinet.

Now, wouldn't it be nice, if we had something like that, a filing
cabinet where we can store large chunks of data and randomly read and
write them in a fast manner?

Oh yes, I remember, it is called a filesystem. Let's use some of those
to store the mail data. It will be s awesome! ;-)


Ok, back being serious: there is nothing wrong with using a RDBMS in the
way it was intented, to store user credentials, quota values, account
settings, forwarding addresses, address book data, bookmarks, etc.


Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.



Re: [Dovecot] Creating an IMAP repo for ~100 users need some advice

2012-03-17 Thread Kaya Saman

On 03/18/2012 12:32 AM, Sven Hartge wrote:

Kaya Samankayasa...@gmail.com  wrote:


Once we get setup this may come in quite handy! Not sure what's going
on currently as everyone above me is still quite set in using an SQL
DB as a mail storage system???

RDBMS where not designed for such a task.  Using a relational database
as a storage method for big chunks of data is very unwise, in my
opinion. It degrades them to just being some sort of filing cabinet.

Now, wouldn't it be nice, if we had something like that, a filing
cabinet where we can store large chunks of data and randomly read and
write them in a fast manner?

Oh yes, I remember, it is called a filesystem. Let's use some of those
to store the mail data. It will be s awesome! ;-)


I think for the serious engineer there's Linux if even more serious 
there's UNIX and for the rest there's MS. Actually as a medical term 
MS is something not that great to have; why does that also equate to 
IT/Computing too ;-P





Ok, back being serious: there is nothing wrong with using a RDBMS in the
way it was intented, to store user credentials, quota values, account
settings, forwarding addresses, address book data, bookmarks, etc.


I agree!

My humble opinion for a personal preference setup in this instance:

FreeBSD 8.2 x64 as base OS
UFS2 running on root drive
Create ZFS pools for storage
Have users mailboxes on the ZFS pools
Enable ZFS caching and snapshots
Dovecot to manage IMAPv4

---

Get rid of MS altogether! wouldn't that be nice

Then start working a really cool implementation of UNIX/Linux only 
infrastructure :-)





Grüße,
Sven.



Regards,

Kaya