date:20130503

On 5/2/2013 8:12 AM, lst_ho...@kwsoft.de wrote:

 IMHO if you say VM than the filesystem inside the guest doesn't matter
 that much. 

Malarky.

 The difference of ext4/xfs are mostly the knowledge and
 adjustability for special (high-end) hardware and the like. With a

XFS doesn't require high end hardware to demonstrate its advantages
over EXT4.  In his LCA 2012 presentation on XFS development, Dave
Chinner showed data from IIRC a 12 disk RAID0 array, which is hardly
high end.  Watch the presentation and note the massive lead XFS has over
EXT4 (and BTRFS) in most areas.  The performance gap is quite
staggering.  You'll see the same performance, and differences, in a VM
or on bare hardware.

http://youtu.be/FegjLbCnoBw

 Hypervisor providing some standard I/O channel and hiding/handling the
 hardware details itself, most of the differences are gone. With this in

Again, malarky.  The parallel performance in XFS resides in multiple
threads and memory structures, b+ trees, and how these are executed and
manipulated, and via the on disk layout of AGs and how they're written
to in parallel.  Virtualization doesn't change nor limit any of this.
The block device driver, not the filesystem, talks through the
hypervisor to the hardware.  No hypervisor imposes limits on XFS
parallelism or performance, nor block device drivers.  Some may be
configured to prioritize IO amongst guests, but that's a different issue
entirely.

Worthy of note here is that nearly all XFS testing performed by the
developers today is done within virtual machines on filesystems that
reside within sparse files atop another XFS filesystem--not directly on
hardware.  According to you, this double layer of virtualization, OS and
filesystem, would further eliminate all meaningful performance
differences between XFS and EXT4.  Yet this is not the case at all
because EXT4 doesn't yet handle sparse files very well, so the XFS lead
increases.

 mind your question should maybe more of what filesystem is more
 Hypervisor friendly. For this i would suspect the simpler the better,
 so i would choose ext4.

Again, malarky.  The hypervisor imposes no limits on filesystem
performance, other than the CPU cycles, scheduling, and RAM overhead of
the hypervisor itself.  I.e. the same things imposed on all aspects of
guest operation.

-- 
Stan

Re: [Dovecot] XFS vs EXT4 for mail storage

On 5/2/2013 7:04 AM, Alessio Cecchi wrote:

 rw,noatime,attr2,delaylog,nobarrier,inode64,noquota
...
 and I'm running it on RHEL 6.4

I assume this is from /proc/mounts?  All of those but for noatime,
nobarrier, and inode64 are defaults.  You've apparently specified these
in /etc/fstab.  noatime is useless as relatime is the default.  Google
XFS relatime vs noatime.

I assume you have a RAID controller or SAN head with [F|B]BWC and have
disabled individual drive write caches of array disks, given you've
disabled journal write barriers.  If drive caches are in fact enabled,
and/or you don't have [F|B]BWC, then journal write barriers need to be
enabled.  If not you're skydiving without a reserve chute.

-- 
Stan

Re: [Dovecot] XFS vs EXT4 for mail storage

On 5/2/2013 8:21 AM, Reindl Harald wrote:
 
 
 Am 02.05.2013 15:17, schrieb Charles Marcus:
 but what I'm still unsure of for a VM
 environment is the delaylog and nobarrier options.

Delaylog is fine for VM guests.  The barrier settings may all simply be
useless because many hypervisors don't pass barriers down the stack from
the guest.  Which means things like fdatasync don't work, not just
journal write barriers.  See:

http://xfs.org/index.php/XFS_FAQ#Q:_Which_settings_are_best_with_virtualization_like_VMware.2C_XEN.2C_qemu.3F

This has negatively affected EXT4 on ESXi, not just XFS.

 Are these recommended/optimal for a VM? Running on ESXi (does it matter what 
 hypervisor ie being used)?
 
 barriers does not help you much or are implicit because you
 have no physical disk under the FS and the underlying storage
 should have battery backed buffers and doe snot need to confirm
 the physical write to the disk to have the data safe

The problem isn't lack of a physical disk under the guest.  The problem
is lack of software support in the hypervisors.  I don't have an answer
as to which versions of ESXi/vSphere/etc, if any, do or do not support
write barriers, fdatasync, etc.  I'm not finding it in their
knowledgebase, though I've not put much effort into it yet.  You'll need
to do some research.

-- 
Stan

Re: [Dovecot] XFS vs EXT4 for mail storage

2013-05-03 Thread lst_hoe02



Zitat von Stan Hoeppner s...@hardwarefreak.com:


On 5/2/2013 8:12 AM, lst_ho...@kwsoft.de wrote:


IMHO if you say VM than the filesystem inside the guest doesn't matter
that much.


Malarky.


If you are going to insult you maybe should write it so non native  
speakers could find it (malarkey).





The difference of ext4/xfs are mostly the knowledge and
adjustability for special (high-end) hardware and the like. With a


XFS doesn't require high end hardware to demonstrate its advantages
over EXT4.  In his LCA 2012 presentation on XFS development, Dave
Chinner showed data from IIRC a 12 disk RAID0 array, which is hardly
high end.  Watch the presentation and note the massive lead XFS has over
EXT4 (and BTRFS) in most areas.  The performance gap is quite
staggering.  You'll see the same performance, and differences, in a VM
or on bare hardware.

http://youtu.be/FegjLbCnoBw


It is not stunningly that a developer of XFS come out with a setup  
where XFS is the fastest at all.



Hypervisor providing some standard I/O channel and hiding/handling the
hardware details itself, most of the differences are gone. With this in


Again, malarky.  The parallel performance in XFS resides in multiple
threads and memory structures, b+ trees, and how these are executed and
manipulated, and via the on disk layout of AGs and how they're written
to in parallel.  Virtualization doesn't change nor limit any of this.
The block device driver, not the filesystem, talks through the
hypervisor to the hardware.  No hypervisor imposes limits on XFS
parallelism or performance, nor block device drivers.  Some may be
configured to prioritize IO amongst guests, but that's a different issue
entirely.


While it might be true that XFS threading and the  
non-blocking/parallel design will gain some benefit, it is no longer  
true for all points regarding disk layout or estimate of i/o  
channels and disk spindles.



Worthy of note here is that nearly all XFS testing performed by the
developers today is done within virtual machines on filesystems that
reside within sparse files atop another XFS filesystem--not directly on
hardware.  According to you, this double layer of virtualization, OS and
filesystem, would further eliminate all meaningful performance
differences between XFS and EXT4.  Yet this is not the case at all
because EXT4 doesn't yet handle sparse files very well, so the XFS lead
increases.


So you have confirmed may suspection that XFS developers will find a  
case where it matters in favour of XFS ;-)
In real world VM deployments most of the time there are vmfs volumes  
(VMWare) underneath, or NTFS (Hyper-V) and in many cases these are  
even taken from some form of SAN device doing its own mapping of fs  
blocks to physical blocks. With this a careful choosen disk layout  
inside the guest doesn't matter at all, if the Hypervisor does not or  
can not map this useful to the hardware.



mind your question should maybe more of what filesystem is more
Hypervisor friendly. For this i would suspect the simpler the better,
so i would choose ext4.


Again, malarky.  The hypervisor imposes no limits on filesystem
performance, other than the CPU cycles, scheduling, and RAM overhead of
the hypervisor itself.  I.e. the same things imposed on all aspects of
guest operation.


You have forgotten that the Hypervisor also provide only a standard  
device API for the I/O channel which limits the possibility to do  
any hardware estimate/optimization inside the guest. So many  
traditional performance tweaks don't work as expected like physical  
block layout or alignment. The more far away in terms of layers you  
are from hardware the more difficult it get to optimize i/o speed with  
the traditional approaches. You can proof this by the myriand of  
benchmarks flying around all have another clear winner dependent on  
who has done the benchmark.


I know your history on insisting your are right in any cases, so this  
is my last post on this subject. Every reader should try to understand  
the differences on his/her own anyway.


Regards

Andreas




smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Dovecot] XFS vs EXT4 for mail storage

2013-05-03 Thread Alessio Cecchi


Il 03/05/2013 10:48, Stan Hoeppner ha scritto:

On 5/2/2013 7:04 AM, Alessio Cecchi wrote:


rw,noatime,attr2,delaylog,nobarrier,inode64,noquota

...

and I'm running it on RHEL 6.4


I assume this is from /proc/mounts?  All of those but for noatime,
nobarrier, and inode64 are defaults.  You've apparently specified these
in /etc/fstab.  noatime is useless as relatime is the default.  Google
XFS relatime vs noatime.

I assume you have a RAID controller or SAN head with [F|B]BWC and have
disabled individual drive write caches of array disks, given you've
disabled journal write barriers.  If drive caches are in fact enabled,
and/or you don't have [F|B]BWC, then journal write barriers need to be
enabled.  If not you're skydiving without a reserve chute.



Thanks Stan, yes the output is from /proc/mounts.

We are running XFS on RAID controller but we havent disabled individual 
drive write caches. So what options suggest in fstab for XFS with non 
high-end RAID/SAN ?


Thanks
--
Alessio Cecchi is:
@ ILS - http://www.linux.it/~alessice/
on LinkedIn - http://www.linkedin.com/in/alessice
Assistenza Sistemi GNU/Linux - http://www.cecchi.biz/
@ PLUG - ex-Presidente, adesso senatore a vita, http://www.prato.linux.it

Re: [Dovecot] dual stack issue (was: Dovecot has no understandable configuration any more)

2013-05-03 Thread Jakob Hirsch

forum, 30.04.2013 16:55:

 Now i have found the solution.
 It seems to help to ask you. ;)
 
 The solution was to change from
 listen = [::]
 to
 listen = *, [::]
 
 There is no explanation for it - just try and error ...

This is not dovecot's fault. See here: http://serverfault.com/a/39561

In short: In Linux, binding to :: means bind to both ipv6 and ipv4.
Setting /proc/sys/net/ipv6/bindv6only to 1 changes this behaviour, and
Debian had this by default in some testing versions of squeeze, as was
recently discussed on the asterisk-users list:
http://lists.digium.com/pipermail/asterisk-users/2013-March/278296.html

Re: [Dovecot] dual stack issue (was: Dovecot has no understandable configuration any more)

On 3.5.2013, at 12.34, Jakob Hirsch j...@plonk.de wrote:

 forum, 30.04.2013 16:55:
 
 Now i have found the solution.
 It seems to help to ask you. ;)
 
 The solution was to change from
 listen = [::]
 to
 listen = *, [::]
 
 There is no explanation for it - just try and error ...
 
 This is not dovecot's fault. See here: http://serverfault.com/a/39561
 
 In short: In Linux, binding to :: means bind to both ipv6 and ipv4.
 Setting /proc/sys/net/ipv6/bindv6only to 1 changes this behaviour, and
 Debian had this by default in some testing versions of squeeze, as was
 recently discussed on the asterisk-users list:
 http://lists.digium.com/pipermail/asterisk-users/2013-March/278296.html

Software can also specify if they want that functionality or not. In Dovecot 
v1.0 [::] may have listened also on IPv4, but since v1.1 [::] has only listened 
on IPv6. So I guess the complaint wasn't really about Dovecot v2.x, but about 
v1.1+.

Hmm. So Debian stable is switching from Dovecot v1.0 - v2.1? Maybe it would be 
worth mentioning this somewhere.

Re: [Dovecot] dual stack issue (was: Dovecot has no understandable configuration any more)

On 3.5.2013, at 12.46, Timo Sirainen t...@iki.fi wrote:

 Hmm. So Debian stable is switching from Dovecot v1.0 - v2.1? Maybe it would 
 be worth mentioning this somewhere.

No, previous stable had v1.2.15, so I guess there won't be any mass moving from 
v1.0 to v2.x.

Re: [Dovecot] XFS vs EXT4 for mail storage

On 5/2/2013 8:17 AM, Charles Marcus wrote:

 man mount doesn't show delaylog, nobarrier or noquota as valid mount
 options... ?

Many XFS mount options are kernel version specific.  Show:
~$ uname -a

Delaylog doesn't exist in recent mount(8) because it's no longer a mount
option, same goes for older mount(8).  Its existence as a mount option
didn't exist for long, WRT distro mount(8) updates.  Since 2.6.39
delaylog is the default, and as of somewhat more recently in the 3.x
tree, the old journal logging code was completely removed from the
source.  Thus there is no longer a delaylog mount option.  The feature
is now hard coded in XFS.

Barriers are crucial to XFS journal, and thus filesystem, reliability.
nobarrier isn't listed in mount(8), though barrier is, which is the
default mode.  Making people look for the switch that disables
barriers forces them to take a learning journey.  Hopefully during this
journey they become educated to the risks of disabling it, before doing
so.  Better reliability through obscurity you might say.  Consider the
horrible rap XFS would have today if everyone and his dog could easily
learn how to disable barriers, then did so on hardware not appropriate
for it.  Yes, exactly, corrupted XFS filesystems littering the landscape
and people screaming what a pile of dogsh|zt XFS is.

WRT noquota, it is the default.  You'd never specify it.  There are 10
quota options at the bottom of the XFS section of mount(8) that one
might want to set.

It is quite irritating, yet surprisingly common, to see XFS users
re-specifying the defaults in their /etc/fstab, because they didn't take
the time to educate themselves properly, and simply copy/pasted from one
of many online XFS tuning guides.  On the XFS list we call these XFS
mis-tuning guides, as nearly all of them contain mostly misinformation.
 Not intentional mind you, but because they just don't know what they're
talking about, or they did but the guide is 5+ years old, and things
have changed.

-- 
Stan

Re: [Dovecot] XFS vs EXT4 for mail storage

On 5/3/2013 4:31 AM, lst_ho...@kwsoft.de wrote:

 If you are going to insult you maybe should write it so non native
 speakers could find it (malarkey).

Sorry Andreas.  I didn't intend that as an insult, merely an expression
of strong disagreement with statements not grounded in facts.

 It is not stunningly that a developer of XFS come out with a setup where
 XFS is the fastest at all.

Dave is even handed with this stuff.  Watch the video.  The pre-delaylog
slides show EXT4 metadata performance really trouncing old XFS by a
*much* larger margin than that of XFS with delaylog over EXT4.  When
delaylog turns the tables the gap is much smaller.  This says more about
how horrible XFS metadata performance was prior to delaylog than how
much better than EXT4 it is today, though it is substantially better
with greater parallelism.
...
 So you have confirmed may suspection that XFS developers will find a
 case where it matters in favour of XFS ;-)

All developers use VMs today for the obvious reason:  It saves so much
time and allows much more work in a given time frame.  Note that for
validation testing of things like barriers they must still use bare
metal since the hypervisors noop disk cache flushes.
...
 I know your history on insisting your are right in any cases, so this is

Then you've obviously missed posts where I've acknowledged making mistakes.

 my last post on this subject. Every reader should try to understand the
 differences on his/her own anyway.

It's never about being right but getting it right.  People require
accurate technical information in order to make technical decisions.  I
provide that when I have the information.  I also try to correct
incomplete, missing, or inaccurate information where I believe it to be
necessary.  You stated that a VM environment eliminates most of the
advantages of any given filesystem, and that's simply not correct.

-- 
Stan

Re: [Dovecot] XFS vs EXT4 for mail storage


On 2013-05-03 1:30 AM, Stan Hoeppner s...@hardwarefreak.com wrote:

 From a filesystem perspective mdbox is little different from maildir as
they both exhibit lots of small random IOs.


Hi Stan. Thanks, was hoping you'd chime in here...

But, I'm confused as to why you'd say this. mdbox supposedly has many 
advantages over maildir, since it is *not* a single file for every email 
(like maildir or sdbox).



My recommendation is to use XFS with the defaults, but add inode64 to
the mount options in /etc/fstab.  This enables the modern allocator
which clusters files around their parent directory within an allocation
group.  It's the default allocator in very recent upstream kernels but
not in most currently shipping distro kernels.  It decreases seek
latency between metadata and file operations, and better manages on disk
space.  In short, XFS will yield superior mail performance to EXT4 in a
multiuser environment.


Thanks very much. I'd already come to a similar conclusion, but was 
starting to have doubts after some of the prior comments. But what you 
say backs up the majority of what I've been reading. It's just difficult 
to judge what you're reading when you aren't a software or hardware 
engineer, just a lowly self-taught sysadmin who still consider himself a 
noob even after doing this for a few years.



There are currently no mail workload tuning docs in the world of XFS
that I'm aware of.  I've been intending to write such a doc for the
XFS.org FAQ for some time but it hasn't happened yet.


Hope you find the time to do it some day... :)

On 2013-05-03 5:54 AM, Stan Hoeppner s...@hardwarefreak.com wrote:

Many XFS mount options are kernel version specific.  Show:
~$ uname -a


Linux myhost 3.7.10-gentoo-r1 #3 SMP Sat Apr 27 10:01:59 EDT 2013 x86_64 
AMD Opteron(tm) Processor 4180 AuthenticAMD GNU/Linux



Delaylog doesn't exist in recent mount(8) because it's no longer a mount
option, same goes for older mount(8).  Its existence as a mount option
didn't exist for long, WRT distro mount(8) updates.  Since 2.6.39
delaylog is the default, and as of somewhat more recently in the 3.x
tree, the old journal logging code was completely removed from the
source.  Thus there is no longer a delaylog mount option.  The feature
is now hard coded in XFS.


Got it, thanks.


Barriers are crucial to XFS journal, and thus filesystem, reliability.
nobarrier isn't listed in mount(8), though barrier is, which is the
default mode.  Making people look for the switch that disables
barriers forces them to take a learning journey.  Hopefully during this
journey they become educated to the risks of disabling it, before doing
so.  Better reliability through obscurity you might say.  Consider the
horrible rap XFS would have today if everyone and his dog could easily
learn how to disable barriers, then did so on hardware not appropriate
for it.  Yes, exactly, corrupted XFS filesystems littering the landscape
and people screaming what a pile of dogsh|zt XFS is.


Got it, thanks again.


WRT noquota, it is the default.  You'd never specify it.  There are 10
quota options at the bottom of the XFS section of mount(8) that one
might want to set.

It is quite irritating, yet surprisingly common, to see XFS users
re-specifying the defaults in their /etc/fstab, because they didn't take
the time to educate themselves properly, and simply copy/pasted from one
of many online XFS tuning guides.  On the XFS list we call these XFS
mis-tuning guides, as nearly all of them contain mostly misinformation.
  Not intentional mind you, but because they just don't know what they're
talking about, or they did but the guide is 5+ years old, and things
have changed.


Ok, so here's my final fstab... thanks again for all of the comments 
(especially yours Stan).


# fs mountpoint   typeopts dump/pass
#
# Non-LVM volumes
#
/dev/sda1   /boot ext4 defaults,noauto  1 2
/dev/sda3   / ext4 defaults 0 1
/dev/sda2   none  swap sw   0 0
/dev/cdrom  /mnt/cdromauto noauto,ro0 0
/dev/fd0/mnt/floppy   auto noauto   0 0
#
# LVM volumes
#
/dev/vg/var /var  xfs  defaults,inode64 0 2
/dev/vg/tmp /tmp  ext2 nodev,noexec,nosuid  0 2
/dev/vg/vtmp/var/tmp  ext2 nodev,nosuid 0 2
/dev/vg/log /var/log  ext2 defaults 0 2
/dev/vg/snaps   /snapsxfs  defaults,inode64 0 2

--

Best regards,

Charles

Re: [Dovecot] XFS vs EXT4 for mail storage


On 2013-05-03 1:14 AM, Michael Weissenbacher m...@dermichi.com wrote:
I've been using XFS for many years now and i strongly recommend it for 
anything besides /boot. Considering a virtual environment i would 
strongly suggest NOT using nobarrier (i.e. use barrier). You can run 
into big trouble should the system ever lose power. In fact the only 
time i ever managed to damage a XFS filesystem on all those years was 
inside a VM and with the nobarrier option on and the UPS died (and yes 
the server had a functioning BBS battery attached to the RAID). The 
delaylog option can be recommended hands-off, since it speeds up 
metadata operations considerably (up to 10 times faster!). And for 
your last quesion, stay with the defaults when doing mkfs.xfs, 
optimizing for stripe width and stipe size and all those other options 
really only make sense on a physical machine.


hth and good luck,
Michael


Thanks Michael. Yes, it helped to solidify my decision to stick with 
xfs. I posted my final fstab just a few minutes ago, which I'm now happy 
with.


--

Best regards,

Charles

Re: [Dovecot] XFS vs EXT4 for mail storage

On 5/3/2013 4:34 AM, Alessio Cecchi wrote:
 Il 03/05/2013 10:48, Stan Hoeppner ha scritto:
 On 5/2/2013 7:04 AM, Alessio Cecchi wrote:

 rw,noatime,attr2,delaylog,nobarrier,inode64,noquota
 ...
 and I'm running it on RHEL 6.4

 I assume this is from /proc/mounts?  All of those but for noatime,
 nobarrier, and inode64 are defaults.  You've apparently specified these
 in /etc/fstab.  noatime is useless as relatime is the default.  Google
 XFS relatime vs noatime.

 I assume you have a RAID controller or SAN head with [F|B]BWC and have
 disabled individual drive write caches of array disks, given you've
 disabled journal write barriers.  If drive caches are in fact enabled,
 and/or you don't have [F|B]BWC, then journal write barriers need to be
 enabled.  If not you're skydiving without a reserve chute.

 
 Thanks Stan, yes the output is from /proc/mounts.
 
 We are running XFS on RAID controller but we havent disabled individual

Which RAID controller?  Does it have BBWC (battery backed write cache)?
 How much cache RAM?

 drive write caches. So what options suggest in fstab for XFS with non
 high-end RAID/SAN ?

Get rid of noatime and use the default, relatime.  Only specify
nobarrier if you have both:

1.  Working BBWC on your RAID card
2.  Individual disk drive caches are disabled
(and preferably a good UPS)

RAID BBWC is worthless if drive caches are still enabled.  This can
corrupt your filesystem if power fails, or the kernel crashes, because
writes to the journal may be lost.

-- 
Stan

Re: [Dovecot] sieve: deleteheader sometimes fails [bug]

2013-05-03 Thread Daniel Piddock

On 02/05/13 20:48, Stephan Bosch wrote:
 On 5/1/2013 9:23 PM, Stephan Bosch wrote:
 On 5/1/2013 4:58 PM, Daniel Piddock wrote:
 On 01/05/13 14:51, Stephan Bosch wrote:

 Hi Stephan,

 I'm totally sure. The mails end up with two Subject headers so the
 addheader line is executing correctly. The bug doesn't even need the
 fileinto and stop lines, I'm not sure why I included them.

 Hmm, bizarre. I cannot reproduce the problem at this end. Could you
 send me the following:

 - dovecot -n output
 - an example message that triggers this behavior

 Could you try to reproduce this with the sieve-test tool?

 /some off-list interaction

 Ok, this should fix it:

 http://hg.rename-it.nl/dovecot-2.1-pigeonhole/rev/0163c45094a3

 Regards,

 Stephan.


That appears to have fixed the issue in my testing.

Thank you for the quick patch.

Dan

Re: [Dovecot] Debian Stable [WAS dual stack issue]

2013-05-03 Thread Ron Leach


On 03/05/2013 10:50, Timo Sirainen wrote:

On 3.5.2013, at 12.46, Timo Sirainent...@iki.fi  wrote:


Hmm. So Debian stable is switching from Dovecot v1.0 -  v2.1? Maybe it would 
be worth mentioning this somewhere.


No, previous stable had v1.2.15, so I guess there won't be any mass moving from 
v1.0 to v2.x.



I've just newly put Debian Stable on a machine.  Stable does have 
1.2.15, as you say.  I've enabled squeeeze-backports, which also 
allows me to install 2.1.7 (dovecot-core 2.1.7-7~bpo50+1, as I write).


(I've not installed either yet, still checking how to bring the 
real-user maildirs across from 1.0.15 without losing the metadata, so 
that all the clients (meaning the human ones) don't have a shock at 
seeing xx,000 'new' mails.  Recently discussed on the list, so plenty 
of advice.)


regards, Ron

Re: [Dovecot] dovecot antispam plugin is not woking

2013-05-03 Thread Trever L. Adams

On 04/30/2013 05:47 AM, Eugene Paskevich wrote:
 On Tue, 30 Apr 2013 12:36:44 +0300, Ravi Kanchan
 ravi_kanchan2...@yahoo.com wrote:

 and plugin configuration is given below:

 protocol imap {
   mail_plugins =  antispam autocreate
 }
 protocol pop3 {
   mail_plugins = antispam autocreate
 }

 I'm not sure this plugin could be used with POP3.
I am pretty sure that it cannot be used with POP3. It requires IMAP.

Trever

Re: [Dovecot] XFS vs EXT4 for mail storage

On 5/3/2013 6:30 AM, Charles Marcus wrote:
 On 2013-05-03 1:30 AM, Stan Hoeppner s...@hardwarefreak.com wrote:
  From a filesystem perspective mdbox is little different from maildir as
 they both exhibit lots of small random IOs.
 
 Hi Stan. Thanks, was hoping you'd chime in here...
 
 But, I'm confused as to why you'd say this. mdbox supposedly has many
 advantages over maildir, since it is *not* a single file for every email
 (like maildir or sdbox).

When I said lots of small random IOs I was leading into the
explanation of why alignment isn't necessary, and actually detrimental
to a mail workload.  It's WRT filesystem alignment to the RAID stripe
that maildir and mdbox are little different.

 Thanks very much. I'd already come to a similar conclusion, but was
 starting to have doubts after some of the prior comments. But what you
 say backs up the majority of what I've been reading. It's just difficult
 to judge what you're reading when you aren't a software or hardware
 engineer, just a lowly self-taught sysadmin who still consider himself a
 noob even after doing this for a few years.

Digesting the inner workings of a filesystem, especially one as complex
and tweakable as XFS, and how they relate to real world workloads, is
not for the faint of heart.  Ironically, today's XFS defaults work
extremely well out of the box for many workloads, including mail.

 There are currently no mail workload tuning docs in the world of XFS
 that I'm aware of.  I've been intending to write such a doc for the
 XFS.org FAQ for some time but it hasn't happened yet.
 
 Hope you find the time to do it some day... :)

I need to get this Dovecot doc thing finished first...

 On 2013-05-03 5:54 AM, Stan Hoeppner s...@hardwarefreak.com wrote:
 Many XFS mount options are kernel version specific.  Show:
 ~$ uname -a
 
 Linux myhost 3.7.10-gentoo-r1 #3 SMP Sat Apr 27 10:01:59 EDT 2013 x86_64
 AMD Opteron(tm) Processor 4180 AuthenticAMD GNU/Linux

3.7, yeah, pretty sure delaylog is no longer an option with this recent
a kernel.
...
 Ok, so here's my final fstab... thanks again for all of the comments
 (especially yours Stan).
 
 # fs mountpoint   typeopts dump/pass

 /dev/vg/var /var  xfs  defaults,inode64 0 2
...
 /dev/vg/snaps   /snapsxfs  defaults,inode64 0 2

I assume /var will hold user mail dirs.  Do /var/ and /snaps reside on
the same RAID array, physical disks?  How about the other filesystems I
snipped?  If you have a large number of filesystems atop the same RAID,
some of them being XFS, this could create a head thrashing problem under
high load increasing latency and thus response times.

Would you mind posting:

~$ xfs_info /dev/vg/var
~$ xfs_info /dev/vg/snaps

-- 
Stan

Re: [Dovecot] XFS vs EXT4 for mail storage

On 5/3/2013 12:14 AM, Michael Weissenbacher wrote:
...
 last quesion, stay with the defaults when doing mkfs.xfs, optimizing for
 stripe width and stipe size and all those other options really only make
 sense on a physical machine.

The potential benefit of alignment is always workload dependent.  If the
hypervisor passes correct RAID geometry to the VM guest and the workload
can benefit from alignment, then you want alignment in the VM filesystem
just as much as on bare metal.

-- 
Stan

Re: [Dovecot] Accessing mail files not owned by imap login user

2013-05-03 Thread Ben Morrow

At  3PM -0700 on  1/05/13 you (PaulM47) wrote:
 
 I have managed to set up a 'maildir' based mail system using fetchmail and
 procmail that delivers sorted mail to folders /var/spool/mail/user1,
 ../user2 etc. 'user1', 'user2' etc. are real users but the mail system is
 run under logged in user 'mailserver'. The mail folders are owned by
 'user1', 'user2' etc. but mail files, when delivered, are owned by
 'mailserver'. I've been unable to find a way to change this behaviour :-(

I'm not sure about the rest of your problem (I'd have to reread the
documentation, since this isn't a setup I use myself, and you can do
that as well as I), but this bit is straightforward. The only way for a
process running as one uid (say, fetchmail running as mailserver) to
create a file owned by another uid is to pass the data to some other
process running either as the target uid or as root.

This means, to get the uids right (which I would strongly recommend),
you need a daemon; fortunately, Dovecot provides an LMTP daemon which
will do just fine. If you set up the Dovecot LMTP server, fetchmail will
deliver to that quite happily, and the delivered mail files will then
have the correct owners. 

You haven't said what you're doing with procmail; if you're doing
anything beyond arranging for the mail to go to the right user you will
need to replace that filtering with sieve. Note that, if you really
must, you can use the sieve-extprograms extension to deliver using
procmail once the LMTP server has switched down to the right uid.

 Dovecot is configured by adding the single line 'mail_location =
 maildir:/var/spool/mail/%u' to 'dovecot.conf', everything else is as
 installed. When I make an imap connection as 'user1' the logon is
 successful, the mail files are moved from ../new/ to ../cur/ but are
 invisible to the imap client. If I 'chown' the mail files to 'user1' they
 instantly become visible to the imap client.

One thing to try is to run imap in preauth mode (see the wiki) under
ktrace/strace/truss/whatever (as the right user, obviously), so you can
see what it's trying to do and where it's failing.

Ben

Re: [Dovecot] XFS vs EXT4 for mail storage


On 2013-05-03 8:34 AM, Stan Hoeppner s...@hardwarefreak.com wrote:

I assume /var will hold user mail dirs.


Yes, in /var/vmail


Do /var/ and /snaps reside on the same RAID array, physical disks?


Yes - vmware host is a Dell R515, with ESXi installed to mirrored 
internal SATA drives, with 8 drives in RAID 10 for all of the VMs. All 
storage is this local storage (no SAN/NAS).


How about the other filesystems I snipped? If you have a large number 
of filesystems atop the same RAID, some of them being XFS, this could 
create a head thrashing problem under high load increasing latency and 
thus response times.


Ouch...

This ESXi host also hosts 2 server 2008R2 vms...


Would you mind posting: ~$ xfs_info /dev/vg/var


 # xfs_info /dev/vg/var
meta-data=/dev/mapper/vg-var isize=256agcount=4, agsize=45875200 
blks

 =   sectsz=512   attr=2
data =   bsize=4096   blocks=183500800, imaxpct=25
 =   sunit=0  swidth=0 blks
naming   =version 2  bsize=4096   ascii-ci=0
log  =internal   bsize=4096   blocks=89600, version=2
 =   sectsz=512   sunit=0 blks, lazy-count=1
realtime =none   extsz=4096   blocks=0, rtextents=0

~$ xfs_info /dev/vg/snaps 


 # xfs_info /dev/vg/snaps
meta-data=/dev/mapper/vg-snaps   isize=256agcount=4, agsize=262144 blks
 =   sectsz=512   attr=2
data =   bsize=4096   blocks=1048576, imaxpct=25
 =   sunit=0  swidth=0 blks
naming   =version 2  bsize=4096   ascii-ci=0
log  =internal   bsize=4096   blocks=2560, version=2
 =   sectsz=512   sunit=0 blks, lazy-count=1
realtime =none   extsz=4096   blocks=0, rtextents=0

Thanks again Stan...

--

Best regards,

Charles

Re: [Dovecot] Accessing mail files not owned by imap login user

2013-05-03 Thread Robert Schetterer

Am 03.05.2013 16:11, schrieb Ben Morrow:
  have managed to set up a 'maildir' based mail system using fetchmail and
 procmail that delivers sorted mail to folders /var/spool/mail/user1,
 ../user2 etc. 'user1', 'user2' etc. are real users but the mail system is
 run under logged in user 'mailserver'. The mail folders are owned by
 'user1', 'user2' etc. but mail files, when delivered, are owned by
 'mailserver'. I've been unable to find a way to change this behaviour

why using procmail, fetchmail,go modern ways

normally its mostly the best to have one extra user vmail group vmail


then use getmail with additional clamdscan and spamc , if you like,
at last give to dovecot deliver which uses sieve ,acl, quota

i.e

getmail.rc

[options]
 verbose = 0
 message_log = /var/log/getmail.log
 read_all = false
 delivered_to = false
 delete = false

 [retriever]
 type = SimplePOP3Retriever
 server = mail.beispiel.com
 username = u...@beispiel.com
 password = .

[filter-1]
allow_root_commands = True
type = Filter_classifier
path = /usr/bin/clamdscan
arguments = (--stdout, --no-summary, -)
exitcodes_drop = (1, )

[filter-2]
allow_root_commands = True
type = Filter_external
path = /usr/bin/spamc
arguments = (-s,  25, -u, us...@server.com)

[destination]
type = MDA_external
path = /usr/lib/dovecot/deliver
arguments = (-d, us...@server.com)
user = vmail
group = vmail

minimum in dovocot.conf something like

...
passdb {
driver = passwd-file
args = scheme=plain-md5 username_format=%n /etc/dovecot/imap.passwd
}
userdb {
driver = passwd-file
args = username_format=%n /etc/dovecot/imap.passwd
default_fields = local_port=%a
}
...

imap.passwd

...
us...@server.com:{plain}pass:vmail:vmail::/usr/local/virtual/us...@server.com::userdb_mail=maildir:~/Maildir
us...@server.com:{plain}pass:vmail:vmail::/usr/localvirtual/us...@server.com::userdb_mail=maildir:~/Maildir
...

...
mkdir /usr/local/virtual/us...@server.com
mkdir /usr/local/virtual/us...@server.com/Maildir
..

chown -R vmail:vmail  /usr/local/virtual/*

notice this is not copy paste stuff, some things may different at your
server and distro !!!

some blog ( only german )

http://sys4.de/de/blog/2013/04/12/abholdienst-fur-mail/


Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

[Dovecot] Idea: POP3 deletion as a flag

GMail doesn't delete mails when POP3 client issues a DELE command for it. 
Instead they just become invisible for future POP3 sessions, but they still 
exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

 - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
 - When DELEting a message, add this flag to the message.
 - When listing messages, skip all messages that have this flag.
 - Also hide this flag from IMAP clients(?)

Thoughts? Probably 20 lines of new code.

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Gedalya

I can certainly see this being very useful in certain environments where 
asking the client to be responsible is not an option (ISP).

Whether or not the flag affects IMAP clients should be configurable.


On 05/03/2013 12:14 PM, Timo Sirainen wrote:

GMail doesn't delete mails when POP3 client issues a DELE command for it. 
Instead they just become invisible for future POP3 sessions, but they still 
exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

  - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
  - When DELEting a message, add this flag to the message.
  - When listing messages, skip all messages that have this flag.
  - Also hide this flag from IMAP clients(?)

Thoughts? Probably 20 lines of new code.

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Stephan Bosch


On 5/3/2013 6:14 PM, Timo Sirainen wrote:

GMail doesn't delete mails when POP3 client issues a DELE command for it. 
Instead they just become invisible for future POP3 sessions, but they still 
exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

  - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
  - When DELEting a message, add this flag to the message.
  - When listing messages, skip all messages that have this flag.


Yes, very useful.


  - Also hide this flag from IMAP clients(?)

Hmm.. why?

Also, from Sieve I can imagine it could be useful to be able to hide 
certain messages from POP3 by setting this flag.


Regards,

Stephan.

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread lst_hoe02



Zitat von Timo Sirainen t...@iki.fi:

GMail doesn't delete mails when POP3 client issues a DELE command  
for it. Instead they just become invisible for future POP3 sessions,  
but they still exist for IMAP/webmail. The same could be implemented  
pretty easily for Dovecot:


 - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
 - When DELEting a message, add this flag to the message.
 - When listing messages, skip all messages that have this flag.
 - Also hide this flag from IMAP clients(?)

Thoughts? Probably 20 lines of new code.


Hmm, this would lead to every increasing mailboxes for POP3 only  
users. What about this


- Use some setting to express that POP3 deleted mail should be IMAP  
mark-as-deleted

- Skip messages marked as deleted with POP3, maybe even as default?

With this on IMAP access you clearly see what is going on. Also if you  
delete Messages by IMAP they won't get downloaded by POP3. This would  
also fit with mail providers doing an IMAP expunge/purge on a fixed  
timebase.


More obvious IMHO

Regards

Andreas




smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Robert Schetterer

Am 03.05.2013 18:14, schrieb Timo Sirainen:
 GMail doesn't delete mails when POP3 client issues a DELE command for it. 
 Instead they just become invisible for future POP3 sessions, but they still 
 exist for IMAP/webmail. The same could be implemented pretty easily for 
 Dovecot:
 
  - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
  - When DELEting a message, add this flag to the message.
  - When listing messages, skip all messages that have this flag.

My vote, do it , if it doesnt break anything else, but a wiki help
should be offered too.

  - Also hide this flag from IMAP clients(?)

wouldnt do this, that may end in user confusion

 
 Thoughts? Probably 20 lines of new code.
 






Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Reinaldo Matukuma

Timo. Hi.
But it will be visible at what folder? Trash? Or will only be visible as a 
conventional message with the delete flag?
I think that this is interesting if the DELE of pop don't let the user confused 
when doing alternate access using a client via pop sometimes/a client via imap 
sometimes.

 From: t...@iki.fi
 Date: Fri, 3 May 2013 19:14:19 +0300
 To: dovecot@dovecot.org
 Subject: [Dovecot] Idea: POP3 deletion as a flag

 GMail doesn't delete mails when POP3 client issues a DELE command for it. 
 Instead they just become invisible for future POP3 sessions, but they still 
 exist for IMAP/webmail. The same could be implemented pretty easily for 
 Dovecot:

  - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
  - When DELEting a message, add this flag to the message.
  - When listing messages, skip all messages that have this flag.
  - Also hide this flag from IMAP clients(?)

 Thoughts? Probably 20 lines of new code.

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Tom Hendrikx

On 03-05-13 18:49, lst_ho...@kwsoft.de wrote:
 
 Zitat von Timo Sirainen t...@iki.fi:
 
 GMail doesn't delete mails when POP3 client issues a DELE command for
 it. Instead they just become invisible for future POP3 sessions, but
 they still exist for IMAP/webmail. The same could be implemented
 pretty easily for Dovecot:

  - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
  - When DELEting a message, add this flag to the message.
  - When listing messages, skip all messages that have this flag.
  - Also hide this flag from IMAP clients(?)

 Thoughts? Probably 20 lines of new code.
 
 Hmm, this would lead to every increasing mailboxes for POP3 only users.
 What about this
 
 - Use some setting to express that POP3 deleted mail should be IMAP
 mark-as-deleted

This conflicts with (many) IMAP user agents that don't use
mark-as-deleted but have some version of 'move to trash folder' instead.
I do agree that it's probably the Right Way (tm) :)

 - Skip messages marked as deleted with POP3, maybe even as default?
 
 With this on IMAP access you clearly see what is going on. Also if you
 delete Messages by IMAP they won't get downloaded by POP3. This would
 also fit with mail providers doing an IMAP expunge/purge on a fixed
 timebase.

Kind regards,
Tom

[Dovecot] feature request: pipe for custom quota dict queries

2013-05-03 Thread Jeff Gustafson

Just a followup to my previous post. I appears that a pipe for quota
queries via dict is not supported right now. Because of the way we group
things we need more flexibility. I'd like to propose that dovecot should
support quota queries just like userdb dict queries. 
My expertise is not C, but I might try to poke around and see if I can
make something work.

...Jeff

Re: [Dovecot] Idea: POP3 deletion as a flag

On 3.5.2013, at 20.30, Tom Hendrikx t...@whyscream.net wrote:

 On 03-05-13 18:49, lst_ho...@kwsoft.de wrote:
 
 Zitat von Timo Sirainen t...@iki.fi:
 
 GMail doesn't delete mails when POP3 client issues a DELE command for
 it. Instead they just become invisible for future POP3 sessions, but
 they still exist for IMAP/webmail. The same could be implemented
 pretty easily for Dovecot:
 
 - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
 - When DELEting a message, add this flag to the message.
 - When listing messages, skip all messages that have this flag.
 - Also hide this flag from IMAP clients(?)
 
 Thoughts? Probably 20 lines of new code.
 
 Hmm, this would lead to every increasing mailboxes for POP3 only users.
 What about this
 
 - Use some setting to express that POP3 deleted mail should be IMAP
 mark-as-deleted
 
 This conflicts with (many) IMAP user agents that don't use
 mark-as-deleted but have some version of 'move to trash folder' instead.
 I do agree that it's probably the Right Way (tm) :)

Depends on what purpose this feature is going to be used for. Many POP3 users 
don't actually want to delete the mails, they just want to download them, and 
their client just happens to delete the mails as well. They might still want to 
use webmail occasionally to find any old mails. For that purpose \Deleted flag 
wouldn't be ideal. But, with this setting it would be possible as well: 
pop3_deleted_flag = \Deleted (although that would need a few more lines of 
code).

 - Skip messages marked as deleted with POP3, maybe even as default?
 
 With this on IMAP access you clearly see what is going on. Also if you
 delete Messages by IMAP they won't get downloaded by POP3. This would
 also fit with mail providers doing an IMAP expunge/purge on a fixed
 timebase.

If mail is expunged from IMAP, it also is expunged from POP3 in any case.

Re: [Dovecot] Idea: POP3 deletion as a flag

On 3.5.2013, at 19.55, Robert Schetterer r...@sys4.de wrote:

 Am 03.05.2013 18:14, schrieb Timo Sirainen:
 GMail doesn't delete mails when POP3 client issues a DELE command for it. 
 Instead they just become invisible for future POP3 sessions, but they still 
 exist for IMAP/webmail. The same could be implemented pretty easily for 
 Dovecot:
 
 - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
 - When DELEting a message, add this flag to the message.
 - When listing messages, skip all messages that have this flag.
 
 My vote, do it , if it doesnt break anything else, but a wiki help
 should be offered too.
 
 - Also hide this flag from IMAP clients(?)
 
 wouldnt do this, that may end in user confusion

Just about zero IMAP clients make IMAP keywords visible to users, so I'm not 
really sure which way to go here. I'm more thinking of it as an internal POP3 
state which doesn't really need to be visible to IMAP clients, but then again I 
guess it wouldn't really hurt either.

Re: [Dovecot] XFS vs EXT4 for mail storage


On 2013-05-03 10:21 AM, Charles Marcus cmar...@media-brokers.com wrote:
How about the other filesystems I snipped? If you have a large number 
of filesystems atop the same RAID, some of them being XFS, this could 
create a head thrashing problem under high load increasing latency 
and thus response times.


Ouch...

This ESXi host also hosts 2 server 2008R2 vms... 


Or did you mean just the other filesystems in this linux VM?

Yes, they are all on the same RAID. The only purpose of the other xfs 
volume - snaps- is to hold snapshots of /var for email backup purposes - 
so, rsnapshot will initiate an LVM snapshot, take the backup, then 
remove the snapshot. /snaps is not used for anything else, and it is the 
only other xfs filesystem.


The others are either ext4 (/ and /boot) or ext2 (/tmp, /var/tmp and 
/var/log)...


--

Best regards,

Charles

Re: [Dovecot] Accessing mail files not owned by imap login user

2013-05-03 Thread PaulM47

Thanks for your replies, I'll need a bit of time to digest what you have
suggested.





--
View this message in context: 
http://dovecot.2317879.n4.nabble.com/Accessing-mail-files-not-owned-by-imap-login-user-tp41890p41965.html
Sent from the Dovecot mailing list archive at Nabble.com.

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Professa Dementia

On 5/3/2013 11:36 AM, Timo Sirainen wrote:

On 3.5.2013, at 19.55, Robert Schetterer r...@sys4.de wrote:

Am 03.05.2013 18:14, schrieb Timo Sirainen:

GMail doesn't delete mails when POP3 client issues a DELE command for it.
Instead they just become invisible for future POP3 sessions, but they still
exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot:

- Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
- When DELEting a message, add this flag to the message.
- When listing messages, skip all messages that have this flag.

My vote, do it , if it doesnt break anything else, but a wiki help
should be offered too.

- Also hide this flag from IMAP clients(?)

wouldnt do this, that may end in user confusion

Just about zero IMAP clients make IMAP keywords visible to users, so I'm not
really sure which way to go here. I'm more thinking of it as an internal POP3
state which doesn't really need to be visible to IMAP clients, but then again I
guess it wouldn't really hurt either.

Be careful about adding features just for the sake of adding features.
Dovecot is already a powerful, but quite complex. This would make it
even more complex and non-standard. POP and IMAP clients do not expect
this behavior from the server. Also, I have never heard anyone request
this feature.

Google has already admitted that they do not delete email, even when
those emails are deleted from the Trash. Google states that emails may
be archived for a minimum of 2 months past when they are deleted -
emails, may, however, be saved forever. You do not know. Google has
lots of storage and their business is marketing, which means selling
people's personal information. Google is large enough that they are
under scrutiny and have to obey EU privacy laws. However, adding this
feature to Dovecot could put smaller email operators in violation of
those laws. Not deleting emails when the client says to, is effectively
archiving, and data that is archived is regulated by EU privacy laws.
Lots of legal problems have arisen as a result of this.

http://arstechnica.com/tech-policy/2011/12/patriot-act-and-privacy-laws-take-a-bite-out-of-us-cloud-business/

http://uk.news.yahoo.com/eu-judges-hear-arguments-google-test-privacy-case-102608029--finance.html

http://www.afterdawn.com/news/article.cfm/2012/03/02/google_in_breach_of_eu_privacy_laws_commissioner_says

The problem is already solved by the POP client. Most have a setting
that allow emails to remain on the server for a period of time. The
default is generally 5 days, but can be set to any value desired. Apple
Mail, Thunderbird and outlook all have this feature. Most smartphones
also have this feature, including the iPhone, Blackberry and Android
devices, and even though the settings are generally not easily
accessible, it defaults to being turned on.

So in conclusion, in my experience, this feature is not one that has
been requested, it makes Dovecot more complex and behave in ways not
expected by the client, and it may have privacy and legal problems. I
would recommend not doing it.

I think it would be better to concentrate on features that people ask
for. As awful as exchange is, the reason people give me why they use it
over a normal email solution is for the groupware features, which
standard POP and IMAP email is still lacking.

Dem

Re: [Dovecot] Idea: POP3 deletion as a flag

On 3.5.2013, at 22.40, Professa Dementia profe...@dementianati.com wrote:

 Be careful about adding features just for the sake of adding features. 
 Dovecot is already a powerful, but quite complex.  This would make it even 
 more complex and non-standard.  POP and IMAP clients do not expect this 
 behavior from the server.  Also, I have never heard anyone request this 
 feature.

Dovecot won't do that by default. And this feature was requested by a large 
customer of ours, so it is needed.. I was mainly wondering about details.

 Google is large enough that they are under scrutiny and have to obey EU 
 privacy laws.  However, adding this feature to Dovecot could put smaller 
 email operators in violation of those laws.  Not deleting emails when the 
 client says to, is effectively archiving, and data that is archived is 
 regulated by EU privacy laws. Lots of legal problems have arisen as a result 
 of this.

I'll add a comment to the setting that enabling it may violate (EU) privacy 
laws.

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread lst_hoe02



Zitat von Timo Sirainen t...@iki.fi:


On 3.5.2013, at 20.30, Tom Hendrikx t...@whyscream.net wrote:


On 03-05-13 18:49, lst_ho...@kwsoft.de wrote:


Zitat von Timo Sirainen t...@iki.fi:


GMail doesn't delete mails when POP3 client issues a DELE command for
it. Instead they just become invisible for future POP3 sessions, but
they still exist for IMAP/webmail. The same could be implemented
pretty easily for Dovecot:

- Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
- When DELEting a message, add this flag to the message.
- When listing messages, skip all messages that have this flag.
- Also hide this flag from IMAP clients(?)

Thoughts? Probably 20 lines of new code.


Hmm, this would lead to every increasing mailboxes for POP3 only users.
What about this

- Use some setting to express that POP3 deleted mail should be IMAP
mark-as-deleted


This conflicts with (many) IMAP user agents that don't use
mark-as-deleted but have some version of 'move to trash folder' instead.
I do agree that it's probably the Right Way (tm) :)


Depends on what purpose this feature is going to be used for. Many  
POP3 users don't actually want to delete the mails, they just want  
to download them, and their client just happens to delete the mails  
as well. They might still want to use webmail occasionally to find  
any old mails. For that purpose \Deleted flag wouldn't be ideal.  
But, with this setting it would be possible as well:  
pop3_deleted_flag = \Deleted (although that would need a few more  
lines of code).




But for simply leave mail on server with POP3 most MUAs have this  
feature anyway today, no? So at least for Thunderbird and Outlook this  
couldn't be the only use-case. Most requests i have seen where because  
of
- Misguided user using POP3 on different devices with the same account  
-- Better use IMAP
- People unable to throw away anything because they are unable to  
decide what is important


For the second case you will end up with multi-gigabyte mailboxes on  
the server with mails no one need anymore. That's why i would prefer  
\Deleted to flag the mail as not as important anymore (can be delted  
if needed) in spirit of the waste basket on desktops.



- Skip messages marked as deleted with POP3, maybe even as default?

With this on IMAP access you clearly see what is going on. Also if you
delete Messages by IMAP they won't get downloaded by POP3. This would
also fit with mail providers doing an IMAP expunge/purge on a fixed
timebase.


If mail is expunged from IMAP, it also is expunged from POP3 in any case.


Yes, that's the intention after all IMHO. You could set your IMAP  
client to expunge on leave or let the mails still around. If the  
provider have some policy to say expunge overy month you have no ever  
growing mailbox and give the provider a hint what can be deleted. For  
archive purpose you still can use IMAP folders.


Regards

Andreas




smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Ajax

On Fri, May 3, 2013 at 3:49 PM, Timo Sirainen t...@iki.fi wrote:

 I'll add a comment to the setting that enabling it may violate (EU)
 privacy laws.


Please consider a stronger, broader statement saying that enabling it may
violate user's expectations of privacy.

I'm not in a position to cite chapter and voice on this now but folks
curious about the practical aspects of these considerations under US law
might find why many prefer POP to IMAP by looking at the recommendations
by, for example, The Electronic Frontier Foundation (EFF) at 
https://ssd.eff.org/.

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Gedalya


On 05/03/2013 04:00 PM, lst_ho...@kwsoft.de wrote:
But for simply leave mail on server with POP3 most MUAs have this 
feature anyway today, no? So at least for Thunderbird and Outlook this 
couldn't be the only use-case. Most requests i have seen where because of
- Misguided user using POP3 on different devices with the same account 
-- Better use IMAP
- People unable to throw away anything because they are unable to 
decide what is important 


I agree. And, what are you going to do? There are times when you need to 
bend over backwards due to the user's uncorrectable technical 
incorrectness. You need to figure out the least expensive (and I don't 
mean money) way to make your service manageable, which doesn't involve 
the impossible - arguing with your users. Your users will come at you 
asking you to undelete the emails they deleted and when you tell them 
that you fulfilled their request to delete the emails, they will say 
that they don't know what you are talking about. And that's the truth. 
The user didn't delete anything. The user agent did. And somehow it all 
ends up being your problem.
If you don't like this feature, don't enable it. But it's useful in some 
situations.

[Dovecot] Dovecot Postfix Quota Policy Service

2013-05-03 Thread Daniel Luttermann

Hallo zusammen,

über den Policy Service von Dovecot zum Abfragen der User Quotas wurde
ja schon diskutiert - auch gibt es eine gute Anleitung zum Einrichten,
nur leider bekomme ich den Policy Server nicht korrekt konfiguriert.

Folgende Fehler werden protokolliert:

May  3 22:00:13 mail postfix/smtpd[17463]: warning: access table 
unix:private/quota-status entry  has empty value
May  3 22:00:42 mail dovecot: quota-status(dan...@dlutt.de): Error: user 
dan...@dlutt.de: Error reading configuration: 
net_connect_unix(/usr/var/run/dovecot/config) failed: Permission denied

Es ist dabei so, dass nach einem Reload der Dienste die erste Mail
noch zugestellt wird, die folgende, eingehende Mail dann jedoch
abgelehnt wird. Nachfolgend die kompletten Logeinträge:

May  3 22:00:12 mail postfix/smtpd[17463]: connect from 
mail-ie0-x236.google.com[2607:f8b0:4001:c03::236]
May  3 22:00:13 mail postfix/smtpd[17463]: warning: access table 
unix:private/quota-status entry  has empty value
May  3 22:00:13 mail postfix/smtpd[17463]: 0EB81172391A: 
client=mail-ie0-x236.google.com[2607:f8b0:4001:c03::236]
May  3 22:00:13 mail postfix/cleanup[17475]: 0EB81172391A: 
message-id=CAKrzS114MaJGND9BxYUiixMMtORmXJqTA3W13B=qar0yw_n...@mail.gmail.com
May  3 22:00:13 mail postfix/qmgr[17429]: 0EB81172391A: 
from=sen...@googlemail.com, size=5409, nrcpt=1 (queue active)
May  3 22:00:13 mail postfix/smtpd[17463]: disconnect from 
mail-ie0-x236.google.com[2607:f8b0:4001:c03::236]
May  3 22:00:13 mail klms-smtp_proxy: Message from sen...@googlemail.com to 
dan...@dlutt.de passed
May  3 22:00:14 mail postfix/smtpd[17482]: connect from localhost[127.0.0.1]
May  3 22:00:14 mail postfix/smtpd[17482]: 00776172391C: 
client=localhost[127.0.0.1], 
orig_client=mail-ie0-x236.google.com[2607:f8b0:4001:c03::236]
May  3 22:00:14 mail postfix/cleanup[17475]: 00776172391C: 
message-id=CAKrzS114MaJGND9BxYUiixMMtORmXJqTA3W13B=qar0yw_n...@mail.gmail.com
May  3 22:00:14 mail postfix/qmgr[17429]: 00776172391C: 
from=sen...@googlemail.com, size=6195, nrcpt=1 (queue active)
May  3 22:00:14 mail postfix/smtpd[17482]: disconnect from localhost[127.0.0.1]
May  3 22:00:14 mail postfix/smtp[17476]: 0EB81172391A: to=dan...@dlutt.de, 
relay=127.0.0.1[127.0.0.1]:10025, delay=1.2, delays=0.54/0.02/0/0.65, 
dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 00776172391C)
May  3 22:00:14 mail postfix/qmgr[17429]: 0EB81172391A: removed
May  3 22:00:14 mail dovecot: lmtp(17484): Connect from local
May  3 22:00:14 mail dovecot: lmtp(17484, dan...@dlutt.de): 
dEa8BE4XhFFMRAAAG4AjPw: sieve: 
msgid=CAKrzS114MaJGND9BxYUiixMMtORmXJqTA3W13B=qar0yw_n...@mail.gmail.com: 
stored mail into mailbox 'INBOX'
May  3 22:00:14 mail dovecot: lmtp(17484): Disconnect from local: Successful 
quit
May  3 22:00:14 mail postfix/lmtp[17483]: 00776172391C: to=dan...@dlutt.de, 
relay=mail.dlutt.de[private/dovecot-lmtp], delay=0.11, 
delays=0.05/0.02/0.01/0.03, dsn=2.0.0, status=sent (250 2.0.0 dan...@dlutt.de 
dEa8BE4XhFFMRAAAG4AjPw Saved)
May  3 22:00:14 mail postfix/qmgr[17429]: 00776172391C: removed
May  3 22:00:36 mail postfix/postscreen[17433]: CONNECT from 
[2607:f8b0:4001:c03::230]:43653 to [2a00:1828:2000:206::2]:25
May  3 22:00:42 mail postfix/postscreen[17433]: PASS NEW 
[2607:f8b0:4001:c03::230]:43653
May  3 22:00:42 mail postfix/smtpd[17463]: connect from 
mail-ie0-x230.google.com[2607:f8b0:4001:c03::230]
May  3 22:00:42 mail dovecot: quota-status(dan...@dlutt.de): Error: user 
dan...@dlutt.de: Error reading configuration: 
net_connect_unix(/usr/var/run/dovecot/config) failed: Permission denied
May  3 22:00:42 mail postfix/smtpd[17463]: NOQUEUE: reject: RCPT from 
mail-ie0-x230.google.com[2607:f8b0:4001:c03::230]: 450 4.7.1 dan...@dlutt.de: 
Recipient address rejected: Internal error occurred. Refer to server log for 
more information.; from=absen...@googlemail.com to=dan...@dlutt.de 
proto=ESMTP helo=mail-ie0-x230.google.com
May  3 22:00:42 mail postfix/smtpd[17463]: disconnect from 
mail-ie0-x230.google.com[2607:f8b0:4001:c03::230]
May  3 22:00:53 mail postfix/postfix-script[17560]: stopping the Postfix mail 
system
May  3 22:00:53 mail postfix/master[17425]: terminating on signal 15
May  3 22:02:51 mail dovecot: master: Warning: Killed with signal 15 (by 
pid=17681 uid=0 code=kill)


Das Problem tritt mit dem unix_listener, aber auch mit dem
inet_listener auf - beide Male beschwert sich Dovecot über den Zugriff
auf den Service config. Ich habe auch mal versucht, die
Berechtigungen für den Service config auf World-Readable zu setzen,
leider hat dies auch nicht geklappt, wobei ich nicht weiß, ob dieser
unbedingt mit dem Problem etwas zu tun hat.

Auch beim Debug-Logging sind leider nicht mehr Infos zum Fehler zu
finden.

Zum Einsatz kommt aktuell Postfix 2.10.0 und Dovecot 2.2.1.

Die Dovecot Quota Konfiguration sieht so aus, wie bei sys4
beschrieben:

service quota-status {
 executable = quota-status -p postfix
 unix_listener /var/spool/postfix/private/quota-status {
 group = postfix
 mode

Re: [Dovecot] Dovecot Postfix Quota Policy Service

2013-05-03 Thread Daniel Luttermann

On 2013-05-03, Daniel Luttermann wrote:

 Hallo zusammen,

 über den Policy Service von Dovecot zum Abfragen der User Quotas wurde
 ja schon diskutiert - auch gibt es eine gute Anleitung zum Einrichten,
 nur leider bekomme ich den Policy Server nicht korrekt konfiguriert.

sorry, this was a question for the german Dovecot mailing list.

--
Daniel

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Simon Brereton

On 3 May 2013 18:14, Timo Sirainen t...@iki.fi wrote:

 GMail doesn't delete mails when POP3 client issues a DELE command for it.
Instead they just become invisible for future POP3 sessions, but they still
exist for IMAP/webmail. The same could be implemented pretty easily for
Dovecot:

  - Add a new setting to enable this: pop3_deleted_flag = $POP3Deleted
  - When DELEting a message, add this flag to the message.
  - When listing messages, skip all messages that have this flag.
  - Also hide this flag from IMAP clients(?)

 Thoughts? Probably 20 lines of new code.

The EU laws not withstanding, I think this is a good thing.  The world
didn't revolve around the EU anymore than it does the US.

If enough people want it, and I can see it's use, even if 90% of my users
won't need it, then put it in.  Warn users to check the regulations of
their jurisdiction before turning it on and off you go..

Simon

Re: [Dovecot] Idea: POP3 deletion as a flag

2013-05-03 Thread Kelsey Cummings


On 2013-05-03 09:14, Timo Sirainen wrote:

GMail doesn't delete mails when POP3 client issues a DELE command for
it. Instead they just become invisible for future POP3 sessions, but
they still exist for IMAP/webmail. The same could be implemented
pretty easily for Dovecot:


How does the usage case by your large customer differ from that allowed 
by the lazy_expunge plugin?

Re: [Dovecot] Idea: POP3 deletion as a flag