Re: [vchkpw] vdelivermail stdout to Dovecot deliver
On Thu, 2009-04-02 at 06:01 -0700, Tom Collins wrote: On Mar 30, 2009, at 7:32 PM, Rick Romero wrote: What I'm trying to work around with this method is to handle user-specific .qmail directives. Dovecot doesn't do that, and that is why I can't full out replace vdelivermail with deliver. What if vpopmail was updated to store a user's .qmail file as domain.com/.qmail-user instead of domain.com/user/.qmail? It seems whatever solution I want to implement has a lot of work involved. In this case it would involve migrating all the existing domain.com/user/.qmail files to domain.com/.qmail-user, and the applications which create/modify them. It's not done through vpopmaild. I think the simplest option is just calling deliver via vdelivermail's existing run_command function (which IMHO does exactly what is needed - it would be exactly the same as calling maildrop via domain.com/user/.qmail), I just have been too busy to really test it. Rick !DSPAM:49d699c132681621912481!
Re: [vchkpw] vdelivermail stdout to Dovecot deliver
On Mar 30, 2009, at 7:32 PM, Rick Romero wrote: What I'm trying to work around with this method is to handle user-specific .qmail directives. Dovecot doesn't do that, and that is why I can't full out replace vdelivermail with deliver. What if vpopmail was updated to store a user's .qmail file as domain.com/.qmail-user instead of domain.com/user/.qmail? Granted, with a lot of users you end up with a huge number of files in the domain directory, and that will have an impact on performance, but it might be a solution to your problem. We'd have to make other changes to the codebase so the alias/forward routines ignored .qmail- user files that belonged to actual POP/IMAP accounts. -Tom !DSPAM:49d4b73032681666196558!
Re: [vchkpw] vdelivermail stdout to Dovecot deliver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rick Romero wrote: vdelivermail already has a 'run_command' function for running .qmail commands, I'm going to try using that next to call deliver. That should solve the chained pipe issue. I'll admit that I'm not very familiar with the issues involved in getting vdelivermail to deliver to Dovecot directly, but I would like to see more support for Dovecot in vpopmail. As soon as there's some preliminary idea of how to get this part working, or some sample code, please let me know so I can look over it. - -- /* Matt Brookings m...@inter7.com GnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAknSJM4ACgkQ6QgvSNlBT3AMSQCfRgRJKWFkAZBWF5yVG8OIupF9 kqYAoITu0i1zvDJfstERjcU2HrviTZqm =3ImN -END PGP SIGNATURE-
Re: [vchkpw] vdelivermail stdout to Dovecot deliver
Ok. This won't work. My test system had all the variables set in the shell, which is why it worked. :( The reason it won't work is that qmail-local is the parent process of both vdelivermail AND deliver. If vdelivermail sets HOME, it does not apply to deliver's environment. :( On the up side, with vdelivermail sending the mail to STDOUT, if you do |/usr/local/vpopmail/bin/vdelivermailstdout | /usr/local/libexec/dovecot/deliver -d $...@$host It should deliver.. I'll try and test this tonite - on my test system I received an error 'email' in my INBOX when $EXT and $HOST didn't exist on my commandline. The caveat being you need to run the dovecot Auth on each machine that does delivery. :/ The other option would be for vdelivermail to call Dovecot's deliver after setting the environment. Programming question - if I write to fd0 (STDOUT), and then exec() a process, will that child process see the data I put in fd0 from the parent? Maybe I'll just try that as well. Rick Quoting Rick Romero r...@havokmon.com: On Wed, 2009-03-11 at 14:19 -0500, Rick Romero wrote: I think it'll work just dandy if vdelivermail set's the HOME variable and writes the email to stdout. I attached a patch, but I think testing this is going to be a pita unless someone has some sort of shell 'vdelivermail' tester ? :O Holy crap it worked. Not only did it compile without error, but it actually worked as expected. The command: cat /home/vpopmail/domains/havokmon.com/rick/Maildir/cur/1236799820.50282.mx.vfemail.net,S=3365:2,S | env -v EXT=rick HOST=havokmon.com HOME=/home/vpopmail/domains/havokmon.com/rick /usr/local/vpopmail/bin/vdelivermailstdout '' r...@havokmon.com Causes the ./vdelivermail (which is compiled to send to STDOUT) to display the email in the terminal If I run: cat /home/vpopmail/domains/havokmon.com/rick/Maildir/cur/1236799820.50282.mx.vfemail.net,S=3365:2,S | env -v EXT=rick HOST=havokmon.com HOME=/home/vpopmail/domains/havokmon.com/rick /usr/local/vpopmail/bin/vdelivermail '' r...@havokmon.com The email will be delivered to my mailbox. So I've got a decent test environment. Now appending deliver to that first command line: cat /home/vpopmail/domains/havokmon.com/rick/Maildir/cur/1236751658.43485.mx.vfemail.net,S=3436:2,S | env EXT=rick HOST=havokmon.com HOME=/home/vpopmail/domains/havokmon.com/rick /usr/local/vpopmail/bin/vdelivermailstdout '' r...@havokmon.com | /usr/local/libexec/dovecot/deliver And it worked too! Wow. I'm blown away. I need a glass of champagne. Not that I didn't think it would work, but that it worked 'pefectly' without throwing an error on the first try. :) I think it took me longer to figure out how to test it in a shell. The only problem I see is the new message starts with a (null). (null)Delivered-To: r...@havokmon.com Now the null occurs whether I use deliver, the original vdelivermail, or the new vdelivermailstdout, so I think its part of the cat. I'll work on it a little more tomorrow, so I can go to bed happy tonite :) Rick !DSPAM:49d1032d32681689686421!
RE: [vchkpw] vdelivermail stdout to Dovecot deliver
I have a question about this. When I first implemented dSPAM I used the same method of nested pipes to handle delivery through .qmail-default. However the problem I ran into was if there was a problem in the first pipe that caused an error mail was lost due to the broken pipe. Is that something that could happen here? Is the pipe intelligent enough to see a failure and notify the previous process? And with regards to the environment variables, if you export them in the parent process shouldn't they be part of the environments of the child processes? Another possibility is piping through maildrop. That's the solution I ended up moving to for dSPAM since it was able to handle errors properly through an exception and xfilter clause. Based on the error codes dspamc sent I could re-queue or do other things. And to ensure that chkuser still functioned properly for bounce-no-mailbox you just setup the .qmail-default like this: | /usr/local/bin/maildrop /etc/maildroprc # bounce-no-mailbox Because chkuser only checks for the existence of bounce-no-mailbox in .qmail-default. It doesn't care about vdelivermail so adding it as a comment works perfectly. I'm not sure if this method would be worth doing in the case of dovecot, but it helped me get around some of the same issues with dSPAM, and ensure that mail was never lost. Regards, Tren -Original Message- From: Rick Romero [mailto:r...@havokmon.com] Sent: Monday, March 30, 2009 10:37 AM To: vchkpw@inter7.com Subject: Re: [vchkpw] vdelivermail stdout to Dovecot deliver Ok. This won't work. My test system had all the variables set in the shell, which is why it worked. :( The reason it won't work is that qmail-local is the parent process of both vdelivermail AND deliver. If vdelivermail sets HOME, it does not apply to deliver's environment. :( On the up side, with vdelivermail sending the mail to STDOUT, if you do |/usr/local/vpopmail/bin/vdelivermailstdout | /usr/local/libexec/dovecot/deliver -d $...@$host It should deliver.. I'll try and test this tonite - on my test system I received an error 'email' in my INBOX when $EXT and $HOST didn't exist on my commandline. The caveat being you need to run the dovecot Auth on each machine that does delivery. :/ The other option would be for vdelivermail to call Dovecot's deliver after setting the environment. Programming question - if I write to fd0 (STDOUT), and then exec() a process, will that child process see the data I put in fd0 from the parent? Maybe I'll just try that as well. Rick Quoting Rick Romero r...@havokmon.com: On Wed, 2009-03-11 at 14:19 -0500, Rick Romero wrote: I think it'll work just dandy if vdelivermail set's the HOME variable and writes the email to stdout. I attached a patch, but I think testing this is going to be a pita unless someone has some sort of shell 'vdelivermail' tester ? :O Holy crap it worked. Not only did it compile without error, but it actually worked as expected. The command: cat /home/vpopmail/domains/havokmon.com/rick/Maildir/cur/1236799820.50282.m x.vfemail.net,S=3365:2,S | env -v EXT=rick HOST=havokmon.com HOME=/home/vpopmail/domains/havokmon.com/rick /usr/local/vpopmail/bin/vdelivermailstdout '' r...@havokmon.com Causes the ./vdelivermail (which is compiled to send to STDOUT) to display the email in the terminal If I run: cat /home/vpopmail/domains/havokmon.com/rick/Maildir/cur/1236799820.50282.m x.vfemail.net,S=3365:2,S | env -v EXT=rick HOST=havokmon.com HOME=/home/vpopmail/domains/havokmon.com/rick /usr/local/vpopmail/bin/vdelivermail '' r...@havokmon.com The email will be delivered to my mailbox. So I've got a decent test environment. Now appending deliver to that first command line: cat /home/vpopmail/domains/havokmon.com/rick/Maildir/cur/1236751658.43485.m x.vfemail.net,S=3436:2,S | env EXT=rick HOST=havokmon.com HOME=/home/vpopmail/domains/havokmon.com/rick /usr/local/vpopmail/bin/vdelivermailstdout '' r...@havokmon.com | /usr/local/libexec/dovecot/deliver And it worked too! Wow. I'm blown away. I need a glass of champagne. Not that I didn't think it would work, but that it worked 'pefectly' without throwing an error on the first try. :) I think it took me longer to figure out how to test it in a shell. The only problem I see is the new message starts with a (null). (null)Delivered-To: r...@havokmon.com Now the null occurs whether I use deliver, the original vdelivermail, or the new vdelivermailstdout, so I think its part of the cat. I'll work on it a little more tomorrow, so I can go to bed happy tonite :) Rick !DSPAM:49d1159e32681919617724!
RE: [vchkpw] vdelivermail stdout to Dovecot deliver
-Original Message- From: Tren Blackburn [mailto:t...@eotnetworks.com] Sent: Monday, March 30, 2009 11:55 AM To: vchkpw@inter7.com Subject: RE: [vchkpw] vdelivermail stdout to Dovecot deliver I have a question about this. When I first implemented dSPAM I used the same method of nested pipes to handle delivery through .qmail-default. However the problem I ran into was if there was a problem in the first pipe that caused an error mail was lost due to the broken pipe. Is that something that could happen here? Is the pipe intelligent enough to see a failure and notify the previous process? And with regards to the environment variables, if you export them in the parent process shouldn't they be part of the environments of the child processes? Another possibility is piping through maildrop. That's the solution I ended up moving to for dSPAM since it was able to handle errors properly through an exception and xfilter clause. Based on the error codes dspamc sent I could re-queue or do other things. And to ensure that chkuser still functioned properly for bounce-no-mailbox you just setup the .qmail-default like this: | /usr/local/bin/maildrop /etc/maildroprc # bounce-no-mailbox Because chkuser only checks for the existence of bounce-no-mailbox in .qmail-default. It doesn't care about vdelivermail so adding it as a comment works perfectly. I'm not sure if this method would be worth doing in the case of dovecot, but it helped me get around some of the same issues with dSPAM, and ensure that mail was never lost. Regards, Tren [Cleaver, J.C.] One option is patching vdelivermail to not see the final delivery through. At a previous location I was at, we had it leave files in Maildir/tmp and trigger an external script. If that exited successfully, vdeliermail left it in Maildir/tmp and exited OK -- presumably leaving that file there for later processing by some other script. If it didn't, back out of the delivery with a tempfail. Ideally, you would have that external script record an entry in a DB queue somewhere. Later on (later meaning a few async seconds later), the queue is polled and the final delivery process does whatever it wants with that guaranteed-safe file, it being responsible for dropping it in Maildir/new when it's complete. If everything goes according to plan, you should be guaranteed not to drop mail. The only drawback is that your final delivery script can't generate a bounce back out through the mail stream automatically, it would have to re-inject. Regards, Japheth Cleaver !DSPAM:49d11a8032681616791409!
RE: [vchkpw] vdelivermail stdout to Dovecot deliver
-Original Message- From: Cleaver, Japheth [mailto:jclea...@soe.sony.com] Sent: Monday, March 30, 2009 12:16 PM To: vchkpw@inter7.com Subject: RE: [vchkpw] vdelivermail stdout to Dovecot deliver -Original Message- From: Tren Blackburn [mailto:t...@eotnetworks.com] Sent: Monday, March 30, 2009 11:55 AM To: vchkpw@inter7.com Subject: RE: [vchkpw] vdelivermail stdout to Dovecot deliver I have a question about this. When I first implemented dSPAM I used the same method of nested pipes to handle delivery through .qmail- default. However the problem I ran into was if there was a problem in the first pipe that caused an error mail was lost due to the broken pipe. Is that something that could happen here? Is the pipe intelligent enough to see a failure and notify the previous process? And with regards to the environment variables, if you export them in the parent process shouldn't they be part of the environments of the child processes? Another possibility is piping through maildrop. That's the solution I ended up moving to for dSPAM since it was able to handle errors properly through an exception and xfilter clause. Based on the error codes dspamc sent I could re-queue or do other things. And to ensure that chkuser still functioned properly for bounce-no-mailbox you just setup the .qmail-default like this: | /usr/local/bin/maildrop /etc/maildroprc # bounce-no-mailbox Because chkuser only checks for the existence of bounce-no-mailbox in .qmail-default. It doesn't care about vdelivermail so adding it as a comment works perfectly. I'm not sure if this method would be worth doing in the case of dovecot, but it helped me get around some of the same issues with dSPAM, and ensure that mail was never lost. Regards, Tren [Cleaver, J.C.] One option is patching vdelivermail to not see the final delivery through. At a previous location I was at, we had it leave files in Maildir/tmp and trigger an external script. If that exited successfully, vdeliermail left it in Maildir/tmp and exited OK -- presumably leaving that file there for later processing by some other script. If it didn't, back out of the delivery with a tempfail. Hmmm...it seems to me like that wouldn't scale very well, and could possibly get messy if you're dealing with a cluster of SMTP delivery nodes. But it's another way for sure. Maildrop doesn't require any hacking of vdelivermail as it's a drop-in replacement for it since vdelivermail is called at the end of maildrop after all the filtering is done inside. This guarantees no loss of mail, without needing extra external processes. Ideally, you would have that external script record an entry in a DB queue somewhere. Later on (later meaning a few async seconds later), the queue is polled and the final delivery process does whatever it wants with that guaranteed-safe file, it being responsible for dropping it in Maildir/new when it's complete. If everything goes according to plan, you should be guaranteed not to drop mail. The only drawback is that your final delivery script can't generate a bounce back out through the mail stream automatically, it would have to re-inject. How much additional overhead is caused due to re-injection over re-queuing? It seems like this requires on quite a few additional pieces that are outside of the qmail delivery path and if any of them break or not work as expected, could delay final delivery infinitely due to messages gathering in the tmp folder. I'm unsure how you would handle remote delivery with this method as it seems to assume the final mailbox is on the same server/cluster as the delivery process. Ideally we shouldn't need *any* of these hacks, maildrop or otherwise. It'd be nice to plug in a different delivery method to qmail. However, that's not really an option, so we build what we can to do what we need, and we all have different needs usually. :) Regards, Tren !DSPAM:49d1215832689616257855!
RE: [vchkpw] vdelivermail stdout to Dovecot deliver
What I'm trying to work around with this method is to handle user-specific .qmail directives. Dovecot doesn't do that, and that is why I can't full out replace vdelivermail with deliver. As for pipes, I see where you're coming from, and it's probably best to not chain pipes, but instead exec the deliver process from within vdelivermail just as it would a user-specific .qmail directive. I was having problems with that, but you've just given me another avenue to try - vdelivermail will exec piped commands, so I may be able to re-use that code. Then your 'piped program failed' action should be no different than when maildrop or procmail is called from .qmail. The problem with the environments is that piping doesn't appear to be creating a child process of the previous command (within .qmail-default). fd1 (I think that's STDOUT) is a persistent file descriptor which each piped process can read, but qmail-local is the actual parent process of everything that runs from .qmail-default - vdelivermail is the parent of everything that runs from ~user/.qmail. I also discovered that I have an .inbox under my domain folder after testing dovecot deliver because $home was set to my domain, which qmail-local does. Unfortuantely it's tried and true :/ Rick On Mon, 2009-03-30 at 11:55 -0700, Tren Blackburn wrote: I have a question about this. When I first implemented dSPAM I used the same method of nested pipes to handle delivery through .qmail-default. However the problem I ran into was if there was a problem in the first pipe that caused an error mail was lost due to the broken pipe. Is that something that could happen here? Is the pipe intelligent enough to see a failure and notify the previous process? And with regards to the environment variables, if you export them in the parent process shouldn't they be part of the environments of the child processes? Another possibility is piping through maildrop. That's the solution I ended up moving to for dSPAM since it was able to handle errors properly through an exception and xfilter clause. Based on the error codes dspamc sent I could re-queue or do other things. And to ensure that chkuser still functioned properly for bounce-no-mailbox you just setup the .qmail-default like this: | /usr/local/bin/maildrop /etc/maildroprc # bounce-no-mailbox Because chkuser only checks for the existence of bounce-no-mailbox in .qmail-default. It doesn't care about vdelivermail so adding it as a comment works perfectly. I'm not sure if this method would be worth doing in the case of dovecot, but it helped me get around some of the same issues with dSPAM, and ensure that mail was never lost. Regards, Tren -Original Message- From: Rick Romero [mailto:r...@havokmon.com] Sent: Monday, March 30, 2009 10:37 AM To: vchkpw@inter7.com Subject: Re: [vchkpw] vdelivermail stdout to Dovecot deliver Ok. This won't work. My test system had all the variables set in the shell, which is why it worked. :( The reason it won't work is that qmail-local is the parent process of both vdelivermail AND deliver. If vdelivermail sets HOME, it does not apply to deliver's environment. :( On the up side, with vdelivermail sending the mail to STDOUT, if you do |/usr/local/vpopmail/bin/vdelivermailstdout | /usr/local/libexec/dovecot/deliver -d $...@$host It should deliver.. I'll try and test this tonite - on my test system I received an error 'email' in my INBOX when $EXT and $HOST didn't exist on my commandline. The caveat being you need to run the dovecot Auth on each machine that does delivery. :/ The other option would be for vdelivermail to call Dovecot's deliver after setting the environment. Programming question - if I write to fd0 (STDOUT), and then exec() a process, will that child process see the data I put in fd0 from the parent? Maybe I'll just try that as well. Rick Quoting Rick Romero r...@havokmon.com: On Wed, 2009-03-11 at 14:19 -0500, Rick Romero wrote: I think it'll work just dandy if vdelivermail set's the HOME variable and writes the email to stdout. I attached a patch, but I think testing this is going to be a pita unless someone has some sort of shell 'vdelivermail' tester ? :O Holy crap it worked. Not only did it compile without error, but it actually worked as expected. The command: cat /home/vpopmail/domains/havokmon.com/rick/Maildir/cur/1236799820.50282.m x.vfemail.net,S=3365:2,S | env -v EXT=rick HOST=havokmon.com HOME=/home/vpopmail/domains/havokmon.com/rick /usr/local/vpopmail/bin/vdelivermailstdout '' r...@havokmon.com Causes the ./vdelivermail (which is compiled to send to STDOUT) to display the email in the terminal If I run: cat /home/vpopmail/domains/havokmon.com/rick/Maildir/cur/1236799820.50282.m x.vfemail.net,S=3365:2,S | env -v EXT=rick HOST
Re: [vchkpw] vdelivermail stdout to Dovecot deliver
On Mon, Mar 30, 2009 at 11:06 PM, Rick Romero r...@havokmon.com wrote: Programming question - if I write to fd0 (STDOUT), and then exec() a process, will that child process see the data I put in fd0 from the parent? Maybe I'll just try that as well. make fd0 a file (using makeseekable) and do lseek (0, 0L, SEEK_SET); Attached is a function which can make stdin seekable. !DSPAM:49d192c032681954743819! makeseekable.c Description: Binary data
RE: [vchkpw] vdelivermail stdout to Dovecot deliver
But regardless of whether the parent is qmail-local or vdelivermail, vdelivermail still provides the same environment variables. I did a test by writing a quick one line shell script that wrote the environment variables to a file and ran it from /home/vpopmail/domains/test.com/test/.qmail via a pipe. It sets all the environment variables that you need: DEFAULT=test DTLINE='Delivered-To: test.com-t...@test.com ' EXT=test EXT2= EXT3= EXT4= HOME=/home/vpopmail/domains/test.com HOST=test.com HOST2=test HOST3=test HOST4=test LOCAL=test.com-test newsender=r...@mx3.eotnetworks.com PWD=/home/vpopmail/domains/test.com/test recipient=test.com-t...@test.com RPLINE='Return-Path: r...@mx3.eotnetworks.com ' sender=r...@mx3.eotnetworks.com UFLINE='From r...@mx3.eotnetworks.com Tue Mar 31 05:17:31 2009 ' USER=test.com _=/home/vpopmail/etc/test.sh You should have all the environment variables you need/want there. And if you need something extra, just have vdelivermail set the environment variable and the pipe in user/.qmail will pick it up, or if using maildrop, use the import command to bring the variables you need in and go crazy. I'd still using something like maildrop that lets you properly handle the pipe as it only requires one pipe; nested pipes cause mail loss when they break due to not being able to properly pass exit codes through the extra pipe (or so it was explained to me when I was fighting with that issue). Maildrop can be setup to handle any circumstance you have to deal with and just defer delivery until the problem is resolved without causing mail loss. Hope that helps, Tren -Original Message- From: Rick Romero [mailto:r...@havokmon.com] Sent: Monday, March 30, 2009 7:32 PM To: vchkpw@inter7.com Subject: RE: [vchkpw] vdelivermail stdout to Dovecot deliver What I'm trying to work around with this method is to handle user-specific .qmail directives. Dovecot doesn't do that, and that is why I can't full out replace vdelivermail with deliver. As for pipes, I see where you're coming from, and it's probably best to not chain pipes, but instead exec the deliver process from within vdelivermail just as it would a user-specific .qmail directive. I was having problems with that, but you've just given me another avenue to try - vdelivermail will exec piped commands, so I may be able to re-use that code. Then your 'piped program failed' action should be no different than when maildrop or procmail is called from .qmail. The problem with the environments is that piping doesn't appear to be creating a child process of the previous command (within .qmail-default). fd1 (I think that's STDOUT) is a persistent file descriptor which each piped process can read, but qmail-local is the actual parent process of everything that runs from .qmail-default - vdelivermail is the parent of everything that runs from ~user/.qmail. I also discovered that I have an .inbox under my domain folder after testing dovecot deliver because $home was set to my domain, which qmail-local does. Unfortuantely it's tried and true :/ Rick On Mon, 2009-03-30 at 11:55 -0700, Tren Blackburn wrote: I have a question about this. When I first implemented dSPAM I used the same method of nested pipes to handle delivery through .qmail- default. However the problem I ran into was if there was a problem in the first pipe that caused an error mail was lost due to the broken pipe. Is that something that could happen here? Is the pipe intelligent enough to see a failure and notify the previous process? And with regards to the environment variables, if you export them in the parent process shouldn't they be part of the environments of the child processes? Another possibility is piping through maildrop. That's the solution I ended up moving to for dSPAM since it was able to handle errors properly through an exception and xfilter clause. Based on the error codes dspamc sent I could re-queue or do other things. And to ensure that chkuser still functioned properly for bounce-no-mailbox you just setup the .qmail-default like this: | /usr/local/bin/maildrop /etc/maildroprc # bounce-no-mailbox Because chkuser only checks for the existence of bounce-no-mailbox in .qmail-default. It doesn't care about vdelivermail so adding it as a comment works perfectly. I'm not sure if this method would be worth doing in the case of dovecot, but it helped me get around some of the same issues with dSPAM, and ensure that mail was never lost. Regards, Tren -Original Message- From: Rick Romero [mailto:r...@havokmon.com] Sent: Monday, March 30, 2009 10:37 AM To: vchkpw@inter7.com Subject: Re: [vchkpw] vdelivermail stdout to Dovecot deliver Ok. This won't work. My test system had all the variables set in the shell, which is why it worked. :( The reason it won't work is that qmail-local is the parent process of both vdelivermail
