Re: [Dovecot] pipe plugin - is anyone interested (or using it)?
Quoting Johannes Berg [EMAIL PROTECTED]: I do think, however, that the two plugins could possibly converge. I don't think they can converge, they work in very different ways, since yours work at the IMAP level while mine works at the storage level. Actually, I work at storage level too now, I modified it. Oh. I missed that. I had a look at your plugin, quite some time ago, and did not think you would have modified it in such a way. I should definitely give it a look; how to you figure out where the mail comes from? No, but I'm using spam as an example of how your plugin could be generalised to work the same as mine with a different configuration. It seems to me that you only consider use cases that depend on the target folder while my generalisation was to make it depend on the pair (source, target). I did not think it would be possible to use the (source,target) pair at the storage level. Now that I understand your proposal a little better, I think it looks like a good idea. Cheers, Nicolas
Re: [Dovecot] pipe plugin - is anyone interested (or using it)?
Quoting Johannes Berg [EMAIL PROTECTED]:
plugin {
(...)
pipe = /var/learn/%u/.spam:spamc -d some.host -L spam
pipe2 = /var/learn/%u/.ham:spamc -d some.host -L ham
(...)
And here I define that any message stored to /var/learn/%u/.spam where
%u is the username (that is learn.spam in the user's IMAP namespace) has
to be piped to the spamc -d some.host -L spam command. And the same
for ham.
The above is why I like my antispam plugin much better: you can push
into *any* folder *from* spam to train as ham.
It certainly is a matter of taste, but that also was the reason why I
did not like your plugin... ;-)
Firstly, if a message is correctly identified as a spam, but I want to
keep it a copy for some reason, copying outside the spam plugin means
learning as ham, so I have to copy it again into the spam plugin to
re-learn it as ham.
Sencondly, I don't know how dspam works, but spamassassin can identify
a message as HAM or SPAM without learning it. If I want to learn a
message correctly identified as spam but not auto-learnt, I have again
to move it out of the spam folder and then back in. The same if a
message is correctly identified by ham but not auto-learnt, I have to
move it to the spam folder and then out of it.
Lastly, I'd prefer my user to make some explicit action to learn a
message rather than have a system that somewhat guesses for them...
I do think, however, that the two plugins could possibly converge.
I don't think they can converge, they work in very different ways,
since yours work at the IMAP level while mine works at the storage
level.
Hence, mine can also be triggered by a delivery with dovecot's
deliver, but when a message is copied or appended, it does not know
where it comes from (but as far as I am concerned, since I don't like
triggering when moving a message out of a folder, it does not matter).
On the other hand, yours only works with IMAP, overriding the COPY
command.
That
would entail being able to specify pipes with two endpoints, right now
you just have the destination folder. In order to do that, the plugin
would have to be configurable like this, with the pipes tried in the
order they are numbered:
# learn into spam as spam
pipe.1 = *SPAM |/learn/as --spam
pipe.2 = UNSURESPAM |/learn/as --spam
# forbid into unsure
pipe.3 = *UNSURE-
# learn from unsure or spam folder as ham
pipe.4 = UNSURE*|/learn/as --ham
pipe.5 = SPAM* |/learn/as --ham
Those only work if you consider messages that are copied from a folder
to a different one. But how do you deal with messages that are
appended from another source (such as messages being delivered or
appended by an IMAP client)?
This would allow writing a simple plugin, having dovecot load it (before
the pipe plugin) and having a function in that plugin called, splitting
the work the antispam plugin would do in that case into two plugins.
I guess that such a plugin is actually more generic than pipe then.
It looks to me that you only consider spam/ham learning as a use for
such a plugin. I think other uses are possible. For example, I
remember someone who was willing to implement an outbox.
Do you have the pipe plugin code in git anywhere I can pull from and see
if I can do such modifications?
No, I have never used git. But you have the tarball.
Cheers,
Nicolas
Re: [Dovecot] pipe plugin - is anyone interested (or using it)?
Hi,
Ben Schumacher wrote:
Could you maybe explain how the configuration works in a little more
detail? For some reason I'm having a bit of trouble wrapping my head
around how this works:
What is the part you don't understand?
namespace private {
separator = .
location = maildir:/var/mail/%u
inbox = yes
hidden = no
}
namespace public {
separator = .
prefix = learn.
location = maildir:/var/learn/%u
inbox = no
hidden = no
}
This is a simple namespace definition, nothing specific to my plugin.
The learn.* part of the namespace is declared public and stored an a
specific location. I did this to avoid counting the learnt messages to
the quota.
plugin {
(...)
pipe = /var/learn/%u/.spam:spamc -d some.host -L spam
pipe2 = /var/learn/%u/.ham:spamc -d some.host -L ham
(...)
And here I define that any message stored to /var/learn/%u/.spam where
%u is the username (that is learn.spam in the user's IMAP namespace) has
to be piped to the spamc -d some.host -L spam command. And the same
for ham.
Hope this helps,
Nicolas
Re: [Dovecot] pipe plugin - is anyone interested (or using it)?
Timo Sirainen wrote: I downloaded it when you first mentioned it, but looks like I never got around to actually looking at it. Yes, something like this would be nice.. One thing at least that should be changed is to configure it using virtual mailbox names instead of full mailbox paths. This isn't really possible with v1.0, but v1.1 should make it pretty easy. You already told me about this last time, but I haven't yet come to read the source code of dovecot 1.1... A few things about the code: save_dest_mail = mail_alloc(ctx-transaction, MAIL_FETCH_PHYSICAL_SIZE, NULL); Quota plugin wanted to know the message's physical size, but your plugin doesn't need it. So you could just use 0 instead of MAIL_FETCH_PHYSICAL_SIZE. Thanks for pointing this. I think write() can return partially written data if the other side isn't reading it fast enough. Using write_full() instead would anyway be better/safer. I just reread my code, and I think my use of write looks safe, since only the amount that was correctly written is skipped with i_stream_skip. Do you think I'm missing something? Do you really need to wait for the executed process to finish? Since this is the only plugin currently creating child processes, I'd setup a SIGCHLD handler and waitpid() there to get rid of the zombie: lib_signals_set_handler(SIGCHLD, TRUE, chld_handler, NULL); Waiting is required if we want the append to fail if the command fails. I guess it should better be a configurable option, don't you think so? Multiple commands are now processed sequentially. I don't know if there's real need for multiple commands, but it would be faster to read the message input just once and send it to all pipes in parallel. I don't think there is real need for multiple commands, but I think it was easier for me to program things like this... ;-) return 0 * WEXITSTATUS(status); returns always 0 :) Hummm... I guess I should avoid drinking alcohol before I program... ;-) It probably should be return WEXITSTATUS(status);; I have no idea why I changed this in such a strange way... I'd also leave stderr as-is for the child process so it could log errors, and for handling syscall failures use: i_fatal(dup2() failed: %m); OK, point taken. One question still: would you consider merging my plugin in dovecot if I ported it to 1.1? Cheers, Nicolas
[Dovecot] pipe plugin - is anyone interested (or using it)?
Hi, A few months ago, I sent a message to this list asking if people would be interested by a pipe plugin (see http://dovecot.org/pipermail/dovecot/2007-May/023005.html ). I received a few answers of people who told they were intersted. A few months later, I sent a new message announcing the availability of my work (see http://dovecot.org/pipermail/dovecot/2007-August/024805.html ). Since then, I have received no feedback... So, is anyone using this plugin? Is anyone using it? Timo, on my first message, you seemed to be interested to merge such a feature (see http://dovecot.org/pipermail/dovecot/2007-May/023055.html ). Are you still interested? All comments are welcome. I think the feature (or a more generic one) would be a worthy addition for dovecot, be it for ham/spam learning, to implement an outbox or anything else I can't think about. All comments about my work are welcome. (Please note that I have not (yet) tried to port it to dovecot 1.1...) Cheers, Nicolas Boullis Ecole Centrale Paris
Re: [Dovecot] pipe plugin
Hi, Charles Marcus wrote: This looks interesting - how does it differ from the DSPAM plug-in? Maybe a generic plug-in could be created that would allow it to be used with any anti-spa, solution (I prefer ASSP myself)... This plugin has nothing spamassassin-specific. Although I don't know any other anti-spam software that spamassassin, I think it should be possible to use this plugin with any of them. This plugin is not even specifi to spam/ham learning, and I mentionned an out box as a possible use for it. As for the differences with the DSPAM plugin, the latter allows to learn a mail as a HAM when it is moved *out* of the SPAM folder. This is done by working at the IMAP level and in a way I consider hackish. (As far as I understand IMAP, moving a message does not exist; you can only copy and then delete the source. Hence it detects copying a message from the SPAM folder, which mean you can't get a copy of a SPAM without learning it as a HAM.) My plugin, is triggered whenever a message is written into a folder. Hence 2 folders are needed for ham/spam learning. But it allows you to trigger on APPENDed messages as well (for messages that come from an external source), or on delivered messages. Feel free to try this plugin with your prefered anti-spam software and to report how fine (or bad) it works for you. Cheers, Nicolas
[Dovecot] pipe plugin
Hi,
Long ago (more than 2 month ago), I told about a pipe plugin I had
written to perform some spam/ham learning (see the attached message).
I'm now proud to say that I can make it available for use by anyone.
You can get it from
http://nicolas.boullis.free.fr/pipe.tar.gz
Note that this is a temporary location. If someone had a good place to
host it, feel free to do (and tell).
I think it would be great if the feature could be merged into dovecot. I
don't care whether my plugin is used or a rewrite is performed...
Cheers,
Nicolas
---BeginMessage---
Hi,
I have written a plugin so that each time a message is added to a
specific box, a program is run and the message is piped into it. Note
that the message is also really added to the box.
I've been using it for nearly 3 month, for spam/ham learning, and have
not had any problem with it. I guess it could also be used to
implement an out box (I remember seeing someone asking for such a
feature on this list). Currently, I use it with a config like this:
(...)
namespace private {
separator = .
location = maildir:/var/mail/%u
inbox = yes
hidden = no
}
namespace public {
separator = .
prefix = learn.
location = maildir:/var/learn/%u
inbox = no
hidden = no
}
(...)
plugin {
(...)
pipe = /var/learn/%u/.spam:spamc -d some.host -L spam
pipe2 = /var/learn/%u/.ham:spamc -d some.host -L ham
(...)
}
Would people be interested by such a feature? If enough people show
some interest, I can try to convince my boss to let me release this
plugin GPL-licensed.
Cheers,
Nicolas Boullis
---End Message---
Re: [Dovecot] modules/imap/lib02_imap_quota_plugin.so: undefined symbol: quota_set
Hi,
Quoting Philipp Kolmann [EMAIL PROTECTED]:
Hi,
I wanted to enable FS quota support in dovecot, and have configured the
following directives:
protocol imap {
mail_plugins = imap_quota
The imap_quota plugin depends on the quota plugin. You should try
mail_plugins = quota imap_quota
Cheers,
Nicolas
Re: [Dovecot] Quota warning not generated
Quoting Doug Council [EMAIL PROTECTED]:
I am testing out the unofficial quota warning patch with Dovecot 1.0.0,
and no matter what values I use for the storage or messages values, the
warning script is never executed.
My plugin settings are:
plugin {
quota = maildir:storage=20480
quota_warning = storage=80%:messages=10 /usr/local/bin/quota-warning.sh
}
(...)
There are more than 10 messages in the mailbox, but the
quota-warning.sh script never gets executed. When using storage=80%,
the maildirsize file has the correct values (not included in this
message), but it never executes the script as well.
Does anyone have any ideas?
As I wrote the patch, I do. ;-)
The warning is triggered when the free space goes below the specified
value, not when the used space goes above. Hence, if you have no limit
on the message count, no warning on the message count will ever be
executed.
Moreover, if the free space is already below the specified value, no
warning is triggered. It is only triggered when a message *brings* the
free space below the specified value.
Hope this helps,
Nicolas Boullis
[Dovecot] How about a pipe plugin?
Hi,
I have written a plugin so that each time a message is added to a
specific box, a program is run and the message is piped into it. Note
that the message is also really added to the box.
I've been using it for nearly 3 month, for spam/ham learning, and have
not had any problem with it. I guess it could also be used to
implement an out box (I remember seeing someone asking for such a
feature on this list). Currently, I use it with a config like this:
(...)
namespace private {
separator = .
location = maildir:/var/mail/%u
inbox = yes
hidden = no
}
namespace public {
separator = .
prefix = learn.
location = maildir:/var/learn/%u
inbox = no
hidden = no
}
(...)
plugin {
(...)
pipe = /var/learn/%u/.spam:spamc -d some.host -L spam
pipe2 = /var/learn/%u/.ham:spamc -d some.host -L ham
(...)
}
Would people be interested by such a feature? If enough people show
some interest, I can try to convince my boss to let me release this
plugin GPL-licensed.
Cheers,
Nicolas Boullis
Re: [Dovecot] quota warning - take 2
Timo Sirainen wrote: On Sat, 2007-03-31 at 23:00 +0200, Nicolas Boullis wrote: - I'm still using system() rather than fork()+exec()+waitpid() since I don't know a good time to run waitpid(). struct child_wait * child_wait_new_with_pid(pid_t pid, child_wait_callback_t *callback, void *context); This would make it easy. That code is already written, but it's not in Dovecot's CVS yet. It would be useful also in dovecot-auth. Originally I was thinking about using the code to replace the waitpid() calls in master, but that change got a bit complex so I forgot about it then. OK, then I guess I should consider it when I do the porting to current CVS HEAD. Right? By the way, any documentation I should read about quota setups, quota rules, and quota roots for IMAP? When I had a look at the quota backport for 1.0, I failed to understand how things work together... Timo, will you consider to add this to your unofficial patches? Added. Does this mean it looks good enough for you and you have nothing to criticize? Wow! Cheers, -- Nicolas Boullis Ecole Centrale Paris
