Re: How do you inject/reprocess an email file into the postfix queue?
On Fri, Feb 03, 2023 at 12:30:32PM -0700, Bryan Arenal wrote: > I have ~1000 emails that were previously sent from a different > platform (but failed due to a system error) and I've been asked to see > if we can re-send them with our Postfix server. Do you have accurate records of the intended envelope recipients of these messages? Note that headers may capture too many or too few recipients, either because some were already delivered and should not be addressed again, or because some (Bcc) are not listed. "Reviving" messages for redelivery can be difficult in general. In specific situations, envelope recovery may be tractable. -- Viktor.
Re: How do you inject/reprocess an email file into the postfix queue?
On 2/3/2023 1:30 PM, Bryan Arenal wrote: Hi there, I have ~1000 emails that were previously sent from a different platform (but failed due to a system error) and I've been asked to see if we can re-send them with our Postfix server. I've looked into postfix's 'sendmail' as well as dropping a test file into /var/spool/postfix/hold and trying to move it into deferred and then flush the queue but neither of those seemed to work. Is this even possible? Thank you in advance! If these files are well-formed plain text messages, you can use the postfix sendmail command to inject them. See the sendmail(1) man page for usage. http://www.postfix.org/sendmail.1.html If these files are some format other than text, you'll need to decide if it's worthwhile to find some way to convert them. If you're lucky there is some existing tool to do that. The postfix queue file format is (intentionally) not documented, and dropping files directly into a queue directory is not supported. -- Noel Jones
Re: How do you inject/reprocess an email file into the postfix queue?
Bryan Arenal: > Hi there, > > I have ~1000 emails that were previously sent from a different > platform (but failed due to a system error) and I've been asked to see > if we can re-send them with our Postfix server. > > I've looked into postfix's 'sendmail' as well as dropping a test file > into /var/spool/postfix/hold and trying to move it into deferred and > then flush the queue but neither of those seemed to work. Is this > even possible? Postfix queue files MUST be written by Postfix. You can't put non-Postfix files there. The only supported interfaces to get a message into the queue are SMTP, QMQP, or the /usr/sbin/sendmail command line. For example, you can use the Postfix sendmail command: /usr/sbin/sendmail -i -f sender recipient < inputfile where inputfile contains RFC2822-formatted text (including MIME). Wietse
How do you inject/reprocess an email file into the postfix queue?
Hi there, I have ~1000 emails that were previously sent from a different platform (but failed due to a system error) and I've been asked to see if we can re-send them with our Postfix server. I've looked into postfix's 'sendmail' as well as dropping a test file into /var/spool/postfix/hold and trying to move it into deferred and then flush the queue but neither of those seemed to work. Is this even possible? Thank you in advance!
Re: Possible to "import" a file into postfix queue?
> On Feb 12, 2021, at 2:43 PM, Wietse Venema wrote: > > sendmail -G -i -f "sender" -- recipient... < file > > The -G will prevent Postfix from rewriting/adding message headers. > It will still prepend a Received: header, but if that breaks DKIM, > then the sender is at fault. > > However, Postfix behavior is defined only for properly formed email. > There is no warranty for messages with lines that exceeed protocol > limits, not followed by , not preceeded by , and > so on. The OP mentioned starting with a queue file, and using "postcat -ebh" to extract its content. With a queue file in hand, one can copy it into the "hold" queue (owner:group "postfix:postfix", mode 0700) and then (assuming nothing else is on "hold"): # postfix stop # postsuper -s # postfix start # postsuper -H ALL If there multiple messages on hold, release just the desired one with: # postqueue -j | jq 'select(.sender == "j...@example.org") | .queue_id)' | postsuper -H - Adjust the jq "select" condition as appropriate. Releasing a queue file from hold does not modify its content. -- Viktor.
Re: Possible to "import" a file into postfix queue?
Tobi: > Wietse, > > On 2/11/21 1:49 PM, Wietse Venema wrote: > > sendmail -f sender recipient... < file > > We could debug and solve the issue just wonder for next time: does > delivery via sendmail command not change the msg by one bit? Especially > no new or changed headers. Because we found our problem to be the exact > position of a \r\n If this is placed one bit more left or right the > error did not occur. That's why we could not reproduce it via swaks > --data @/file as this adds at least one header which then pushed the > problematic \r\n to an UN-problematic position. > Anyway the error was in our external content filter and is fixed now. > Was an error that had been undetected in our content-filter for more > than 10 years :-) sendmail -G -i -f "sender" -- recipient... < file The -G will prevent Postfix from rewriting/adding message headers. It will still prepend a Received: header, but if that breaks DKIM, then the sender is at fault. However, Postfix behavior is defined only for properly formed email. There is no warranty for messages with lines that exceeed protocol limits, not followed by , not preceeded by , and so on. Wietse
Re: Possible to "import" a file into postfix queue?
Wietse, On 2/11/21 1:49 PM, Wietse Venema wrote: > sendmail -f sender recipient... < file We could debug and solve the issue just wonder for next time: does delivery via sendmail command not change the msg by one bit? Especially no new or changed headers. Because we found our problem to be the exact position of a \r\n If this is placed one bit more left or right the error did not occur. That's why we could not reproduce it via swaks --data @/file as this adds at least one header which then pushed the problematic \r\n to an UN-problematic position. Anyway the error was in our external content filter and is fixed now. Was an error that had been undetected in our content-filter for more than 10 years :-) Cheers tobi
Re: Possible to "import" a file into postfix queue?
On Thu, Feb 11, 2021 at 07:49:30AM -0500, Wietse Venema wrote: > > So we thought it could be possible to somehow "import" such an affected > > message directly into postfix queue to leave out swaks which may fix > > something in the message. Is there such a postfix command to "import" a > > file as message directly into postfix queues? > > sendmail -f sender recipient... < file Make that: sendmail -i -f "sender" -- recipient... < file just in case: * The file contains lines starting with "." * If the sender is expanded from a variable, make sure to double-quote it, and use: '-f "$sender"', rather than '-f"$sender"', just in case the sender address is empty. * The first recipient address starts with a "-" - Such a recipient is bounced, unless "allow_min_user" is changed. from its default value. -- Viktor.
Re: Possible to "import" a file into postfix queue?
Tobi: > So we thought it could be possible to somehow "import" such an affected > message directly into postfix queue to leave out swaks which may fix > something in the message. Is there such a postfix command to "import" a > file as message directly into postfix queues? sendmail -f sender recipient... < file Wietse
Possible to "import" a file into postfix queue?
Hi all we have a very strange issue with our postfix and its external content-filter. To debug this we wanted to take an affected message (postcat -bhq QID >/tmp/file) and use swaks with --data command to send it again to postfix and through the content filter. But the swaks message does NOT trigger our issue and therefore is properly processed. So we cannot reproduce it over swaks but see in postfix logs that every try from postfix to deliver the message to the filter gets 450 from filter. This 450 is intentional as we have some very rare cornercases where mail gets broken somewhere in the content filter, so to avoid this cases our filter returns 450 to postfix for such mail. So we thought it could be possible to somehow "import" such an affected message directly into postfix queue to leave out swaks which may fix something in the message. Is there such a postfix command to "import" a file as message directly into postfix queues? Thanks for any idea as we really need to be able to reproduce it or else debugging will be very hard :-) Cheers tobi
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On 10/28/20 2:38 PM, Wietse Venema wrote: One possible way out is to skip the Postfix sendmail command, and to use a "mini sendmail" program that submits mail via SMTP. adding an msmtp sender as the VirusAction script in clamav milter, though a bit of 'extra', certainly is the simplest. easy to implement, and seems to behave under my testing -- no more perms errors. will watch for awhile.
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On 10/28/20 2:38 PM, Wietse Venema wrote: One possible way out is to skip the Postfix sendmail command, and to use a "mini sendmail" program that submits mail via SMTP. i've typically got msmtp rattling around. Obviously that will fail when Postfix is down. noted. not ideal, but not critical. It is theoretically possible to build a 'submit via authenticated SMTP' mode into the Postfix sendmail command, and/or to save unsubmitted mail to ~/dead.letter, but both add complexity (new bugs) and introduce new failure modes (more bugs). i generally don't like the idea of adding more failure points. i'm working to remove as much cruft as possible. adding the notification script already is counterproductive to that goal. seems to me there should be a way to give that extra script -- called by a service milter that _itself_ seems to have all its needed perms for its 'normal' ops -- the correct rights & perms to talk to sendmail. from _outside_ of postfix. which seems like not-a-postfix matter.
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
PGNet Dev: > my clamav-milter.conf includes > > VirusAction /usr/local/etc/clamav/scripts/virus-alert.sh > > where that script _does_ invoke sendmail. > > found this process > > ps ax | grep virus > 15670 ?S 0:00 /bin/bash > /usr/local/etc/clamav/scripts/virus-alert.sh > Sanesecurity.Jurlbl.Auto.85a586.UNOFFICIAL 4DLaEC42Q6z2Q > ngocquyen...@gmail.com myu...@example.com Engineer Tee Shirt & Science Tee > Shirt > Tue, 27 Oct 2020 17:29:06 -0700 > > hanging around, obviously dealing with a spammy "Engineer Tee Shirt & Science > Tee Shirt" send. > > i killed it & verified that the current stream of > > warning: mail_queue_enter: create file maildrop/731085.15673: > Permission denied > > in postfix logs indeed stops. > > so, seems like a likely bet! One possible way out is to skip the Postfix sendmail command, and to use a "mini sendmail" program that submits mail via SMTP. Obviously that will fail when Postfix is down. It is theoretically possible to build a 'submit via authenticated SMTP' mode into the Postfix sendmail command, and/or to save unsubmitted mail to ~/dead.letter, but both add complexity (new bugs) and introduce new failure modes (more bugs). Wietse
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On 10/28/20 11:36 AM, PGNet Dev wrote: On 10/28/20 11:30 AM, Viktor Dukhovni wrote: You might start with: # grep -r NoNewPrivileges /etc/systemd i couldn't find any direct, relevant postdrop/maildrop, or NoNewPrivileges, references i chased sendmail usage instances instead. i've clamav-milter in my milter chain on this box. my clamav-milter.conf includes VirusAction /usr/local/etc/clamav/scripts/virus-alert.sh where that script _does_ invoke sendmail. found this process ps ax | grep virus 15670 ?S 0:00 /bin/bash /usr/local/etc/clamav/scripts/virus-alert.sh Sanesecurity.Jurlbl.Auto.85a586.UNOFFICIAL 4DLaEC42Q6z2Q ngocquyen...@gmail.com myu...@example.com Engineer Tee Shirt & Science Tee Shirt Tue, 27 Oct 2020 17:29:06 -0700 hanging around, obviously dealing with a spammy "Engineer Tee Shirt & Science Tee Shirt" send. i killed it & verified that the current stream of warning: mail_queue_enter: create file maildrop/731085.15673: Permission denied in postfix logs indeed stops. so, seems like a likely bet! clamav-milter does NOT have the NoNewPermissions=true set, and therefore defaults to NoNewPermissions=false That^ 'virus-alert.sh' -- exec'ing 'via' clamav-milter, obviously included in my postfix config -- needs to be given appropriate permissions for sending via sendmail.postfix. The question is: what/where is the correct method, so as NOT to break my postfix security model in some foolish manner ?
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On 10/28/20 11:30 AM, Viktor Dukhovni wrote: You might start with: # grep -r NoNewPrivileges /etc/systemd and all other directories with systemd unit files. yup. already done. nothing --other than the now "=false" (need to double check if that's the same as _removing_ it ) in pflogsumm.service -- under /etc/systemd there _are_ the standard instances in /usr/lib/systemd/system/systemd-*, but none that seem immediately relevant. i know I've done this "to myself" with the install; i've just _never_ had the problem with any DIY-installed postfix*. so, atm, elusive :-/
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On Wed, Oct 28, 2020 at 11:22:55AM -0700, PGNet Dev wrote: > On 10/28/20 10:32 AM, Viktor Dukhovni wrote: > > Indeed a process with "no_new_privs" will not be able to run sendmail(1) > > to submit new email. > > noted. > > that said, this _just_ reappeared here, > >postfix/postdrop[15673]: warning: mail_queue_enter: create file > maildrop/678088.15673: Permission denied > > so _not_ (just) pflogsumm :-/ > > ack that Postfix is just the msngr. > > now to find what _other_ process is triggering this! You might start with: # grep -r NoNewPrivileges /etc/systemd and all other directories with systemd unit files. -- Viktor.
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On 10/28/20 10:32 AM, Viktor Dukhovni wrote: Indeed a process with "no_new_privs" will not be able to run sendmail(1) to submit new email. noted. that said, this _just_ reappeared here, postfix/postdrop[15673]: warning: mail_queue_enter: create file maildrop/678088.15673: Permission denied so _not_ (just) pflogsumm :-/ ack that Postfix is just the msngr. now to find what _other_ process is triggering this!
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On Wed, Oct 28, 2020 at 06:19:10PM +0100, Bastian Blank wrote: > > Barring interference from SELinux or AppArmour, ... this should not > > happen unless file permissions change. > > Maybe this was true ten years ago, but it is not longer. The OP even > mentioned something called "no new privileges", which is described only > one tiny internet search away at > https://www.kernel.org/doc/html/latest/userspace-api/no_new_privs.html. The "..." after "SELinux or AppArmour" is inclusive of other controls that modify expected POSIX semantics. Indeed a process with "no_new_privs" will not be able to run sendmail(1) to submit new email. > It seems that you have to learn a lot about the security controls that > a modern Linux provides. A less patronising tone would be welcome... -- Viktor.
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
Hi Viktor On Wed, Oct 28, 2020 at 01:00:35PM -0400, Viktor Dukhovni wrote: > On Wed, Oct 28, 2020 at 09:01:38AM -0700, PGNet Dev wrote: > > Oct 28 15:02:40 svr019 postfix/postdrop[64624]: warning: > > mail_queue_enter: create file maildrop/553726.64624: Permission denied > > Oct 28 15:02:45 svr019 postfix/postdrop[32688]: warning: > > mail_queue_enter: create file maildrop/766615.32688: Permission denied > Barring interference from SELinux or AppArmour, ... this should not > happen unless file permissions change. Maybe this was true ten years ago, but it is not longer. The OP even mentioned something called "no new privileges", which is described only one tiny internet search away at https://www.kernel.org/doc/html/latest/userspace-api/no_new_privs.html. And yes, this flag is exactly what can cause this: it disables suid/sgid, so the maildrop process runs without it and of course can't write into the maildrop directory. > Correct file permissions are set > via "postfix set-permissions" (at package install time) and should not > change thereafter. The OP even showed that the permissions are correct, but you did not cite it. > Of course if SELinux et. al, decide to intervene, then you have to > fix the relevant settings. It seems that you have to learn a lot about the security controls that a modern Linux provides. Regards, Bastian -- I object to intellect without discipline; I object to power without constructive purpose. -- Spock, "The Squire of Gothos", stardate 2124.5
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On Wed, Oct 28, 2020 at 10:13:23AM -0700, PGNet Dev wrote: > > For reference, on my system: > > > > $ postconf setgid_group > > setgid_group = maildrop > > $ ls -ld /var/spool/postfix/maildrop > > drwx-wx--- 2 postfix maildrop 2 Oct 28 12:52 > > /var/spool/postfix/maildrop > > $ ls -l /usr/local/sbin/postdrop > > -rwxr-sr-x 1 root maildrop 41656 Oct 25 03:44 > > /usr/local/sbin/postdrop > > here (fwiw, this is Fedora32-packaged Postfix et al), minor diff > >postconf setgid_group > !! setgid_group = postdrop The specific group name is not important. It just has to be the same group for the executable and the directory. >ls -ld /var/spool/postfix/maildrop > drwx-wx--- 2 postfix postdrop 4.0K Oct 28 08:45 > /var/spool/postfix/maildrop/ > >ls -l `which postdrop` > -rwxr-sr-x 1 root postdrop 25K Aug 31 02:52 /usr/sbin/postdrop* These look OK. > as set by a relatively new install -- i.e., _not_ manually by me, > >ls -ald / /var /var/spool /var/spool/postfix > > dr-xr-xr-x. 22 root root 4.0K Sep 6 10:59 // > drwxr-xr-x. 21 root root 4.0K Oct 27 20:36 /var/ > drwxr-xr-x 14 root root 4.0K Oct 25 19:05 /var/spool/ > drwxr-xr-x 16 root root 4.0K Aug 31 02:52 /var/spool/postfix/ That "x." suggests some extended POSIX ACLs on / and /var, but they're likely OK. You can double-check these. Otherwise, you can check that no kernel settings disable setgid execution. > so far, on this install, 'everything works' ... except for the > eventual/random appearance of those^ complaints. You'll need to figure why the EPERM happens. Postfix is just the messenger. -- Viktor.
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On 10/28/20 10:00 AM, Viktor Dukhovni wrote: On Wed, Oct 28, 2020 at 09:01:38AM -0700, PGNet Dev wrote: Oct 28 15:02:40 svr019 postfix/postdrop[64624]: warning: mail_queue_enter: create file maildrop/553726.64624: Permission denied Oct 28 15:02:45 svr019 postfix/postdrop[32688]: warning: mail_queue_enter: create file maildrop/766615.32688: Permission denied Barring interference from SELinux or AppArmour, neither in-place/enabled here atm ... this should not happen unless file permissions change. Correct file permissions are set via "postfix set-permissions" (at package install time) yup. and, also tried re-exec'ing here. so far, no curative effect by itself. and should not change thereafter. that's what i've always seen to date. and why i suspect it's something _else_ i've done that's monkeying with it. For reference, on my system: $ postconf setgid_group setgid_group = maildrop $ ls -ld /var/spool/postfix/maildrop drwx-wx--- 2 postfix maildrop 2 Oct 28 12:52 /var/spool/postfix/maildrop $ ls -l /usr/local/sbin/postdrop -rwxr-sr-x 1 root maildrop 41656 Oct 25 03:44 /usr/local/sbin/postdrop here (fwiw, this is Fedora32-packaged Postfix et al), minor diff postconf setgid_group !! setgid_group = postdrop ls -ld /var/spool/postfix/maildrop drwx-wx--- 2 postfix postdrop 4.0K Oct 28 08:45 /var/spool/postfix/maildrop/ ls -l `which postdrop` -rwxr-sr-x 1 root postdrop 25K Aug 31 02:52 /usr/sbin/postdrop* Provided that setgid bit and group of the postdrop executable allows it to write to /var/spool/postfix/maildrop, and all parent directories have "x" for all users, all should work. If /, /var, /var/spool, /var/spool/postfix are not world-accessible (x bit for "other"), or if /var/spool/maildrop is not group-wx, or postdrop is not setgid, or has the wrong group, then things don't work. as set by a relatively new install -- i.e., _not_ manually by me, ls -ald / /var /var/spool /var/spool/postfix dr-xr-xr-x. 22 root root 4.0K Sep 6 10:59 // drwxr-xr-x. 21 root root 4.0K Oct 27 20:36 /var/ drwxr-xr-x 14 root root 4.0K Oct 25 19:05 /var/spool/ drwxr-xr-x 16 root root 4.0K Aug 31 02:52 /var/spool/postfix/ so far, on this install, 'everything works' ... except for the eventual/random appearance of those^ complaints. after changing that NoNewPerms bit in the pflogsumm.sh, I'm seeing no further errors. now that I've clearly jinxed it ... i'll keep watch some more.
Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
On Wed, Oct 28, 2020 at 09:01:38AM -0700, PGNet Dev wrote: > Oct 28 15:02:40 svr019 postfix/postdrop[64624]: warning: > mail_queue_enter: create file maildrop/553726.64624: Permission denied > Oct 28 15:02:45 svr019 postfix/postdrop[32688]: warning: > mail_queue_enter: create file maildrop/766615.32688: Permission denied Barring interference from SELinux or AppArmour, ... this should not happen unless file permissions change. Correct file permissions are set via "postfix set-permissions" (at package install time) and should not change thereafter. For reference, on my system: $ postconf setgid_group setgid_group = maildrop $ ls -ld /var/spool/postfix/maildrop drwx-wx--- 2 postfix maildrop 2 Oct 28 12:52 /var/spool/postfix/maildrop $ ls -l /usr/local/sbin/postdrop -rwxr-sr-x 1 root maildrop 41656 Oct 25 03:44 /usr/local/sbin/postdrop Provided that setgid bit and group of the postdrop executable allows it to write to /var/spool/postfix/maildrop, and all parent directories have "x" for all users, all should work. If /, /var, /var/spool, /var/spool/postfix are not world-accessible (x bit for "other"), or if /var/spool/maildrop is not group-wx, or postdrop is not setgid, or has the wrong group, then things don't work. Of course if SELinux et. al, decide to intervene, then you have to fix the relevant settings. -- Viktor.
postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"
on a new, from-distro-pkgs install of Postfix, i've noted an intermittent perms problem it'll run just fine for quite awhile, then I start seeing a steady stream of ... Oct 28 15:02:40 svr019 postfix/postdrop[64624]: warning: mail_queue_enter: create file maildrop/553726.64624: Permission denied Oct 28 15:02:45 svr019 postfix/postdrop[32688]: warning: mail_queue_enter: create file maildrop/766615.32688: Permission denied ... I don't intentionally/explicitly 'do' anything with maildrop/postdrop, and am not yet sure what triggers the problem. spool/queue perms are an obvious 1st suspect ... as installed, atm, i've ls -al /var/spool/postfix/ total 64K drwxr-xr-x 16 rootroot 4.0K Aug 31 02:52 ./ drwxr-xr-x 14 rootroot 4.0K Oct 25 19:05 ../ drwx-- 2 postfix root 4.0K Oct 28 07:51 active/ drwx-- 2 postfix root 4.0K Oct 26 16:43 bounce/ drwx-- 2 postfix root 4.0K Aug 31 02:52 corrupt/ drwx-- 18 postfix root 4.0K Sep 1 15:35 defer/ drwx-- 18 postfix root 4.0K Sep 1 15:35 deferred/ drwx-- 2 postfix root 4.0K Aug 31 02:52 flush/ drwx-- 2 postfix root 4.0K Aug 31 02:52 hold/ drwx-- 2 postfix root 4.0K Oct 28 08:00 incoming/ drwx-wx--- 2 postfix postdrop 4.0K Oct 27 17:00 maildrop/ drwxr-xr-x 2 rootroot 4.0K Oct 28 08:04 pid/ drwx-- 2 postfix root 4.0K Oct 28 08:04 private/ drwx--x--- 2 postfix postdrop 4.0K Oct 28 08:04 public/ drwx-- 2 postfix root 4.0K Aug 31 02:52 saved/ drwx-- 2 postfix root 4.0K Oct 17 04:54 trace/ ls -al `which postfix` `which postqueue` -rwxr-xr-x 1 root root 21K Aug 31 02:52 /usr/sbin/postfix* -rwxr-sr-x 1 root postdrop 25K Aug 31 02:52 /usr/sbin/postqueue* I'm mostly sure (still checking) that my 'culprit' is an overly aggressive 'pflogsumm.service'; changing, [Unit] Description=Daily pflogsumm report [Service] Type=oneshot ExecStart=/usr/local/etc/postfix/scripts/pflogsumm-report.sh KillMode=process LockPersonality=yes MemoryDenyWriteExecute=yes - NoNewPrivileges=yes + NoNewPrivileges=no PrivateDevices=yes PrivateTmp=yes ProtectControlGroups=yes ProtectHome=yes ProtectKernelLogs=yes ProtectKernelModules=yes ProtectKernelTunables=yes ProtectSystem=strict ReadWritePaths=/var/spool/postfix/ RestrictNamespaces=yes RestrictRealtime=yes seems to do the trick; at least, I've not seen the log perm errors since I made the change; _do_ need to watch for a bit. I'd cribbed the .service from posts online, so don't have full confidence in the hardening settings yet. Is there, generally, any expected harm/risk to a Postfix setup's security/perms model with that^ change, - NoNewPrivileges=yes + NoNewPrivileges=no that should better be handled in Postfix configs?
Re: Postfix queue
I am generally using postfix-logwatch for tracking log files. However, I was thinking about something that is working in real time, scripts can be run by cron in some period time. Then the best solution could be write some syslog phraser or redirect syslog to some application eg. written in Java to interpret log file in realtime. Maybe someone has a better idea?
Re: Postfix queue
I am generalny using postfix-logwatch for tracking log files. However, I was thinking about something that is working in real time, scripts can be run by cron in some period time. Then the best could be write some syslog phraser or redirect syslog to some application eg. written in Java to interpret log file in realtime. 2018-02-12 17:59 GMT+01:00 Wietse Venema <wie...@porcupine.org>: > j.emerlik: > > I would like have policy service that will be able to write do data base > > some information eg. when exactly message was sent, message ID, DSN if > > soemthing goes wrong. That means it should be working with Postfix queue. > > You can use "postqueue -j" to get a machine-readable queue listing > with arrival time, and why mail is still in the queue. > > However for historical information of past deliveries you will have > to rely on Postfix logs. > > Wiietse >
Re: Postfix queue
j.emerlik: > I would like have policy service that will be able to write do data base > some information eg. when exactly message was sent, message ID, DSN if > soemthing goes wrong. That means it should be working with Postfix queue. You can use "postqueue -j" to get a machine-readable queue listing with arrival time, and why mail is still in the queue. However for historical information of past deliveries you will have to rely on Postfix logs. Wiietse
Re: Postfix queue
Viktor Dukhovni: > > > > On Feb 12, 2018, at 10:06 AM, j.emerlik <j.emer...@gmail.com> wrote: > > > > It is possible to write some policy service that will be working with > > postfix queue ? > > No. That's a bad idea anyway. To track message flow, parse the logs. The closest that comes to this is a daemon that responds to transport_maps queries. If you must do that, I suggest using a socketmap table. http://www.postfix.org/transport.5.html http://www.postfix.org/socket_table.5.html But, it will drain performance. Wietse
Re: Postfix queue
On 12.02.2018 16:44, j.emerlik wrote: > I would like have database and there information : Message ID, Sent > Date (or last date of send trying), DSN, number of send attempts, > Mail_From, RCPT_TO. That type of information should be extracted from the Postfix logs, as existing tools like 'pflogsumm' do. -Ralph
Re: Postfix queue
I would like have database and there information : Message ID, Sent Date (or last date of send trying), DSN, number of send attempts, Mail_From, RCPT_TO. It would be helpful to create statistics or check exacly what happened with the messages sent, eg, six months ago. 2018-02-12 16:22 GMT+01:00 Ralph Seichter <m16+post...@monksofcool.net>: > On 12.02.2018 16:06, j.emerlik wrote: > > > I would like have policy service that will be able to write do data > > base some information eg. when exactly message was sent, message ID, > > DSN if soemthing goes wrong. That means it should be working with > > Postfix queue. > > That's not really a specific description of your needs or design goals, > so you can expect answers to be similarly vague. ;-) Have you looked > into postqueue(1) and/or showq(8) yet? > > -Ralph > >
Re: Postfix queue
> On Feb 12, 2018, at 10:06 AM, j.emerlik <j.emer...@gmail.com> wrote: > > It is possible to write some policy service that will be working with postfix > queue ? No. That's a bad idea anyway. To track message flow, parse the logs. -- Viktor.
RE: Postfix queue
Hi, I would think you could write a script to do what you need ?
Here is one I use that is in Python.
[root@mta3 alf02013]# Summary
Usage: Summary -s -h {-|POSTFIX_LOG} [ POSTFIX_LOG .. ]
Summarize postfix mail log. Gzipped files are OK.
Print one line for each delivered email, with these columns
TIME_RECEIVED TIME_SENT ELAPSED QUEUEID SOURCE_IP
AUTHENTICATE_USER FINAL_STATUS FROM_ADDR TO_ADDRS
OPTIONS
-h Print column headers
-s Include email subject (if in Postfix log)
-ANGELO FAZZINA
ITS Service Manager:
Spam and Virus Prevention
Mass Mailing
G Suite/Gmail
ang...@uconn.edu
University of Connecticut, ITS, SSG, Server Systems
860-486-9075
From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org]
On Behalf Of j.emerlik
Sent: Monday, February 12, 2018 10:07 AM
To: postfix-users@postfix.org
Subject: Postfix queue
Lately I wrote in python postfix policy service that can do something for me
what I want.
Now I am thinking about next service butI don't know maybe it is not possible.
That is my question:
There is posssible write some service similar to eg.
check_policy_service unix:private/policy-spf
It is possible to write some policy service that will be working with postfix
queue ?
I would like have policy service that will be able to write do data base some
information eg. when exactly message was sent, message ID, DSN if soemthing
goes wrong. That means it should be working with Postfix queue.
But policy services can be configured with smtpd_sender_restrictions and
smtpd_recipient_restrictions.
It 's possible configure some policy service with postfix queue ?
Regards,
MattX
Re: Postfix queue
On 12.02.2018 16:06, j.emerlik wrote: > I would like have policy service that will be able to write do data > base some information eg. when exactly message was sent, message ID, > DSN if soemthing goes wrong. That means it should be working with > Postfix queue. That's not really a specific description of your needs or design goals, so you can expect answers to be similarly vague. ;-) Have you looked into postqueue(1) and/or showq(8) yet? -Ralph
Postfix queue
Lately I wrote in python postfix policy service that can do something for me what I want. Now I am thinking about next service butI don't know maybe it is not possible. That is my question: There is posssible write some service similar to eg. check_policy_service unix:private/policy-spf It is possible to write some policy service that will be working with postfix queue ? I would like have policy service that will be able to write do data base some information eg. when exactly message was sent, message ID, DSN if soemthing goes wrong. That means it should be working with Postfix queue. But policy services can be configured with smtpd_sender_restrictions and smtpd_recipient_restrictions. It 's possible configure some policy service with postfix queue ? Regards, MattX
Re: Postfix Queue
On Sat, Mar 12, 2016 at 01:14:17AM -0800, Robert Wolfe wrote: > "I read that postifx will retry emails for 5 days before it will stop and > email will be removed from the queue. That is too much. https://tools.ietf.org/html/rfc5321#section-4.5.4.1 Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days. It MAY be appropriate to set a shorter maximum number of retries for non- delivery notifications and equivalent error messages than for standard messages. The parameters to the retry algorithm MUST be configurable. This advice should be taken seriously. However, in some environments it is also important to notify the sender that their email did not get through in a timely manner. Therefore, reducing that timeout to 2-3 days is not unreasonable, in combination with http://www.postfix.org/postconf.5.html#delay_warning_time # Warn sender of trouble after 2 hours, continue to retry for 2 days # delay_warning_time = 2h maximal_queue_lifetime = 2d > "I want to set email delivery to be retried only 3 times before it is > removed from the queue permanently. And maximum time email should be in > queue should be only 3 hours. This is a bad idea, and if the OP's organization is spamming and cares not for the delivery of their email, they can use some other MTA. People who ask for short queue lifetime generally have some other problem they've failed to describe or address. The best thing is to ask them what their real problem is, though it is possible they have OCD and want to tune the queue lifetime "just because"... Postfix will only give up after a failed delivery that occurs at least maximal_queue_lifetime after the message is received. If the mail queue has a multi-day backlog, this could in principle be multiple days after the configured maximal_queue_lifetime. > How do I set or configure the maximum delivery retry times/duration for > emails in queue." The relevant parameters are http://www.postfix.org/postconf.5.html#maximal_queue_lifetime http://www.postfix.org/postconf.5.html#bounce_queue_lifetime http://www.postfix.org/postconf.5.html#delay_warning_time http://www.postfix.org/postconf.5.html#maximal_backoff_time For queue tuning advice, users should consult: http://www.postfix.org/QSHAPE_README.html -- Viktor.
Postfix Queue
Hi all. Saw this posted on the iRedMail forum and thought I would post it here in the hopes that I could give a decent response to this poster's question: "I read that postifx will retry emails for 5 days before it will stop and email will be removed from the queue. That is too much. "I want to set email delivery to be retried only 3 times before it is removed from the queue permanently. And maximum time email should be in queue should be only 3 hours. How do I set or configure the maximum delivery retry times/duration for emails in queue."
Re: Postfix: queue size and replies
Peter: > Hi lads (and girls), > > Is there an easy way to set the auto-replies (or answers, not sure how > are they called) in postfix like greylist does "come back later", or > "try the next available MX gateway" when my queue size reaches a certain > amount? It often happens that my primary MX servers are busy crunching > emails meanwhile other servers are slacking off. That would require a policy server (postfwd, policyd, ...). Wietse
Postfix: queue size and replies
Hi lads (and girls), Is there an easy way to set the auto-replies (or answers, not sure how are they called) in postfix like greylist does "come back later", or "try the next available MX gateway" when my queue size reaches a certain amount? It often happens that my primary MX servers are busy crunching emails meanwhile other servers are slacking off. P.
Re: Preventing Postfix queue from being filled
ah got it (always late). Maximal and bounce queue liftemes. On 2014-12-20 09:16, Istvan Prosinger wrote: Dear friends, Sometimes I have similar issues in the que: (connect to hotel-patrizietta.ch[185.53.177.20]:25: Connection timed out) i...@hotel-patrizietta.ch etc, etc, etc. As far as I can tell, Postfix would keep on trying to send these mails for some time. Is there a way to limit this time or the number of attempts? Regards, Istvan
Re: Preventing Postfix queue from being filled
Istvan Prosinger skrev den 2014-12-20 09:16: Is there a way to limit this time or the number of attempts? what if ?, you should not maintain remote problem resolving, since its not your postfix that have problems but if you really like to test, you can change queue lifetime from default 5d, if you run a well configured backup mx then it could be extended not more limited, and then use failback relay on the non backup mta so first attempt wold be normal try, but in case of fails it would be moved to your own backup mx with longer time on redeliver try and queue life time
Re: Preventing Postfix queue from being filled
Istvan Prosinger skrev den 2014-12-20 09:24: ah got it (always late). Maximal and bounce queue liftemes. it would be your fail if you set them lower then default
dropping messages to postfix queue
Hello, I have an application that during its operation it generates lots of RFC-822 format emails. My task is to send them to a single remote email address, and I've decided that I rely on postfix to do the job. I wouldn't use a regular smtp chat with the postfix smtpd daemon, because I don't want to block the application until smtpd receives the message. I know that I may pass the message to the sendmail binary, too, but forking a program every time I need to send an email looks pretty resource heavy to me. My next idea is to just modify the application, and drop (with the link() syscall) the message directly to postfix's queue directory, and hopefully qmgr or something notices that aha! I've got a new message to deliver. In this setup postfix wouldn't get emails from other sources, eg. from the network, just from either the sendmail binary through popen(), or the direct copy to the queue directory method. So my question is whether it's possible, and if so, then how? Best regards, Albi
Re: dropping messages to postfix queue
Kov?cs Albert: I wouldn't use a regular smtp chat with the postfix smtpd daemon, because I don't want to block the application until smtpd receives the message. There is no need to block the application. Use parallism. My next idea is to just modify the application, and drop (with the link() syscall) the message directly to postfix's queue directory, and hopefully qmgr or something notices that aha! I've got a new message to deliver. In that case, you are on your own, with zero support. Translation: unlike the supported SMTP and sendmail iinterfaces that are maintained until eternity, your direct-to-queue submission method will break without any warning as Postfix evolves. Wietse
Re: dropping messages to postfix queue
Hello Wietse, thanks for the clarification, I'd stick to a stable and supported method. On the QSHAPE_REAME page you say that try to keep the volume of local mail injection to a moderate level. Can you give me a rough estimation on moderate level for such an environment where the only source of messages is the local injection? I mean messages per second or similar. And is there a way to force the smtp utility to send multiple messages through an already established connection? Ie. instead of QUIT, continue with another MAIL FROM command (the recipient address is fixed and the same for all messages). Or the postfix smtp program is smart enough and does so whenever it's possible without instructing it? Albi On Wednesday, October 23, 2013 12:45 PM, Wietse Venema wie...@porcupine.org wrote: Kov?cs Albert: I wouldn't use a regular smtp chat with the postfix smtpd daemon, because I don't want to block the application until smtpd receives the message. There is no need to block the application. Use parallism. My next idea is to just modify the application, and drop (with the link() syscall) the message directly to postfix's queue directory, and hopefully qmgr or something notices that aha! I've got a new message to deliver. In that case, you are on your own, with zero support. Translation: unlike the supported SMTP and sendmail iinterfaces that are maintained until eternity, your direct-to-queue submission method will break without any warning as Postfix evolves. Wietse
Re: dropping messages to postfix queue
Kov?cs Albert: Hello Wietse, thanks for the clarification, I'd stick to a stable and supported method. Thanks. And I with I had a better answer. Speaking which, Postfix supports QMQP. This protocol is simpler than SMTP, designed by Dan Bernstein, and is based on netstrings. Netstrings are simple to implement and are already available as libraries for pupular programming languages. http://cr.yp.to/proto/qmqp.html But this does nothing to remove the file system bottleneck. Can you give me a rough estimation on moderate level for such an environment where the only source of messages is the local injection? I mean messages per second or similar. That depends totally on the file system, and the size of messages. You can test this with the Postfix smtp-source utility. That is part of the source code distribution (untar and make) and normally not installed. As required by SMTP (and other) rules, Postfix must fsync() each message before claiming that it has accepted the message. On rust (i.e. rotating disks) this takes many milliseconds because it involves updates of data blocks, inode blocks, bitmap blocks, most or all of which are in different locations on the drive. Even if you have a large write cache it will eventually fill up and heads will have to move and wait for sectors to fly by. On silicon (ramdisk, SSD) this takes less time because there are no head movement or rotational latency. And is there a way to force the smtp utility to send multiple messages through an already established connection? Ie. instead of QUIT, continue with another MAIL FROM command (the recipient address is fixed and the same for all messages). Or the postfix smtp program is smart enough and does so whenever it's possible without instructing it? As required by the SMTP protocol Postfix accepts multiple messages over one SMTP connection. But that does nothing to remove the file system bottleneck. Wietse
Re: dropping messages to postfix queue
On 10/23/2013 6:22 AM, Kovács Albert wrote: On the QSHAPE_REAME page you say that try to keep the volume of local mail injection to a moderate level. Can you give me a rough estimation on moderate level for such an environment where the only source of messages is the local injection? I mean messages per second or similar. This means the input rate must not exceed the output rate for an extended period. Disk throughput will likely be the bottleneck; SSD or ramdisk will help. And is there a way to force the smtp utility to send multiple messages through an already established connection? This is already done automatically, but may not make much difference in a local network where the TCP setup time is short. http://www.postfix.org/CONNECTION_CACHE_README.html -- Noel Jones
Re: question about postfix queue scheduler
On 06/08/2013 08:17 PM, Wietse Venema wrote: Jeroen Geilman: On 06/04/2013 02:20 PM, Erwan David wrote: On Tue, Jun 04, 2013 at 01:44:46PM CEST, Tom Hendrikx t...@whyscream.net said: On 06/04/2013 01:22 PM, Antonio Guti?rrez Mayoral wrote: Hi Wietse, Yes, its a solution, but these emails should be delivered in bussines-time :-( (it doesnt matter if it takes 2 hours... but in bussiness time...) thank you so much! You could run a script as a cronjob that queues x messages when the active queue contains (100 minus x) messages (where 100 is an arbitrary number). This means that all mails on HOLD trickle out as quick as possible, while not overloading the active queue... It means when the queue has 100 messages, you stop sending anything ? You could check the headers for identifying features (maybe the list ID, or a subject part, or...whatever works), and instantly DEFER them. This will put all messages in the deferred queue, guaranteeing they won't choke incoming: if the deferred queue is not empty, one message will be taken from incoming and deferred, in turn. Currently the queue manager can group recipients into jobs when they share the same queue file, and uses that to prevent a limited number of many-recipient messages from blocking later email with fewer recipients. The fix would be to group recipients into jobs based on the sender attribute (or size, or whatever) and apply similar logic to prevent a limited messages from one sender from blocking later email from other senders (or or to prevent large messages from blocking later messages that are smaller in size). However if one sender manages to saturate the queue then it will take time before other email gets a chance to be scheduled. Wietse I thought the queue manager took one message each from deferred and incoming if deferred is not empty, keyed on the destination next-hop (resulting in one virtual queue per destination); this allows one to manipulate the way messages are queued by limiting the number of recipients per message. If he can just use a transport with a single-recipient limit to send his newsletter to, that would take care of the blockage, wouldn't it ? The queue manager doesn't combine multiple queue messages AFAIK, so even if there are hundreds of large single-recipient messages with the same next-hop in the deferred queue, it would only take one message (plus its one recipient) every time, and a single message from incoming after that. -- J.
Re: question about postfix queue scheduler
If he can just use a (sender-dependent) transport to send his newsletter to, that would take care of the blockage, wouldn't it ? Yes, provided that he does not saturate the active queue. There is, however, no need to cripple this transport with single-recipient deliveries. If one delivery transaction can transfer more than one recipient, then that will shorten the time that this mail occupies the queue. Wietse
Re: question about postfix queue scheduler
Antonio Gutiérrez Mayoral skrev den 2013-06-04 13:22: Maybe you can automatically HOLD all his mail and then automatically release all his mail in the evening. Yes, its a solution, but these emails should be delivered in bussines-time :-( (it doesnt matter if it takes 2 hours... but in bussiness time...) cant etrn be used here ? let remote server keep on hold forever until local send etrn to get mails sent ? -- senders that put my email into body content will deliver it to my own trashcan, so if you like to get reply, dont do it
Re: question about postfix queue scheduler
On 06/04/2013 02:20 PM, Erwan David wrote: On Tue, Jun 04, 2013 at 01:44:46PM CEST, Tom Hendrikx t...@whyscream.net said: On 06/04/2013 01:22 PM, Antonio Gutiérrez Mayoral wrote: Hi Wietse, Yes, its a solution, but these emails should be delivered in bussines-time :-( (it doesnt matter if it takes 2 hours... but in bussiness time...) thank you so much! You could run a script as a cronjob that queues x messages when the active queue contains (100 minus x) messages (where 100 is an arbitrary number). This means that all mails on HOLD trickle out as quick as possible, while not overloading the active queue... It means when the queue has 100 messages, you stop sending anything ? You could check the headers for identifying features (maybe the list ID, or a subject part, or...whatever works), and instantly DEFER them. This will put all messages in the deferred queue, guaranteeing they won't choke incoming: if the deferred queue is not empty, one message will be taken from incoming and deferred, in turn. -- J.
Re: question about postfix queue scheduler
Jeroen Geilman: On 06/04/2013 02:20 PM, Erwan David wrote: On Tue, Jun 04, 2013 at 01:44:46PM CEST, Tom Hendrikx t...@whyscream.net said: On 06/04/2013 01:22 PM, Antonio Guti?rrez Mayoral wrote: Hi Wietse, Yes, its a solution, but these emails should be delivered in bussines-time :-( (it doesnt matter if it takes 2 hours... but in bussiness time...) thank you so much! You could run a script as a cronjob that queues x messages when the active queue contains (100 minus x) messages (where 100 is an arbitrary number). This means that all mails on HOLD trickle out as quick as possible, while not overloading the active queue... It means when the queue has 100 messages, you stop sending anything ? You could check the headers for identifying features (maybe the list ID, or a subject part, or...whatever works), and instantly DEFER them. This will put all messages in the deferred queue, guaranteeing they won't choke incoming: if the deferred queue is not empty, one message will be taken from incoming and deferred, in turn. Currently the queue manager can group recipients into jobs when they share the same queue file, and uses that to prevent a limited number of many-recipient messages from blocking later email with fewer recipients. The fix would be to group recipients into jobs based on the sender attribute (or size, or whatever) and apply similar logic to prevent a limited messages from one sender from blocking later email from other senders (or or to prevent large messages from blocking later messages that are smaller in size). However if one sender manages to saturate the queue then it will take time before other email gets a chance to be scheduled. Wietse
question about postfix queue scheduler
Hi all! My name is Antonio I am currently managing a system with postfix+ spamassassin and Maia. First of all, sorry for my bad english. I would like to ask a question. Once in a month, more or less, I have a user who sends a LOT of mails (mailing) to a group of recipients. This list could have about 2000 recipients. The email has two pdfs, maybe 2 MB per email. When this user sends this email, maybe the queue hold about 2000 requests, today about 300 MB. The problem is that the queue starts to deliver the email and the other email which sends the other users, is not being processed until ALL the emails for this user has been processed (FIFO queue) So, the only way I have to deal with is HOLD all messages for this user, with postsuper -h and requeue them step by step, maybe 100 emails per 5 minutes. I dont know if there is a way for improve this. The main problem is the email can not be stopped, the emails have to in/out normally. And, I cannot say to this user that I cant send this emails... :-) Maybe the only solution is install another smtp or an marketing-email app ? Any suggestions? Thanks! -- -- Antonio Gutiérrez Mayoral aguti...@gmail.com
Re: question about postfix queue scheduler
Maybe you can automatically HOLD all his mail and then automatically release all his mail in the evening. Wietse
Re: question about postfix queue scheduler
* Wietse Venema postfix-users@postfix.org: Maybe you can automatically HOLD all his mail and then automatically release all his mail in the evening. I even have a script for that... -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Re: question about postfix queue scheduler
Hi Wietse, Yes, its a solution, but these emails should be delivered in bussines-time :-( (it doesnt matter if it takes 2 hours... but in bussiness time...) thank you so much! 2013/6/4 Wietse Venema wie...@porcupine.org Maybe you can automatically HOLD all his mail and then automatically release all his mail in the evening. Wietse -- -- Antonio Gutiérrez Mayoral aguti...@gmail.com
Re: question about postfix queue scheduler
On 06/04/2013 01:22 PM, Antonio Gutiérrez Mayoral wrote: Hi Wietse, Yes, its a solution, but these emails should be delivered in bussines-time :-( (it doesnt matter if it takes 2 hours... but in bussiness time...) thank you so much! You could run a script as a cronjob that queues x messages when the active queue contains (100 minus x) messages (where 100 is an arbitrary number). This means that all mails on HOLD trickle out as quick as possible, while not overloading the active queue... -- Tom
Re: question about postfix queue scheduler
On Tue, Jun 04, 2013 at 01:44:46PM CEST, Tom Hendrikx t...@whyscream.net said: On 06/04/2013 01:22 PM, Antonio Gutiérrez Mayoral wrote: Hi Wietse, Yes, its a solution, but these emails should be delivered in bussines-time :-( (it doesnt matter if it takes 2 hours... but in bussiness time...) thank you so much! You could run a script as a cronjob that queues x messages when the active queue contains (100 minus x) messages (where 100 is an arbitrary number). This means that all mails on HOLD trickle out as quick as possible, while not overloading the active queue... It means when the queue has 100 messages, you stop sending anything ?
Regenerate or move postfix queue
Hi. I'm running out of space for my postfix's spool directory and I'm planning to create a new 30GB partition and mount it over the current spool dir (/var/spool/postfix). How do I re-generate the queue after mounting the new partition and having an emtpy /var/spool/postfix/ ? In squid, as an example, exists squid -Z which recreates postfix's queue. There is something similar in postfix? Maybe I should check CentOS postfix's post-install scripts instead (/etc/postfix/post-install)? Another option would be (after stopping postfix), move the old queue contents to the new queue, but I'm not sure if that would be OK (in qmail, it couldn't be done because each message in the queue depends on the fs i-node). Thanks. PS: # ls -ld /var/spool/postfix/ drwxr-xr-x 16 root root 4096 oct 1 2009 /var/spool/postfix/ # ls -l /var/spool/postfix/ total 1736 drwx-- 2 postfix root 32 jun 21 09:27 active drwx-- 2 postfix root 4096 jun 21 06:54 bounce drwx-- 2 postfix root 4096 ago 14 2008 corrupt drwx-- 18 postfix root 4096 jul 2 2010 defer drwx-- 18 postfix root 4096 jul 2 2010 deferred drwx-- 2 postfix root 4096 ago 14 2008 flush drwx-- 2 postfix root 4096 ago 14 2008 hold drwx-- 2 postfix root 761856 jun 21 09:25 incoming drwx-wx--- 2 postfix postdrop 16384 jun 21 09:25 maildrop drwxr-xr-x 2 rootroot 4096 jul 2 2010 pid drwx-- 2 postfix root 4096 may 26 12:11 private drwx--x--- 2 postfix postdrop 4096 may 26 12:11 public drwx-- 2 postfix root 4096 ago 14 2008 saved drwx-- 2 postfix root 4096 ago 14 2008 trace
Re: Regenerate or move postfix queue
Santiago Romero: Hi. I'm running out of space for my postfix's spool directory and I'm planning to create a new 30GB partition and mount it over the current spool dir (/var/spool/postfix). How do I re-generate the queue after mounting the new partition and having an emtpy /var/spool/postfix/ ? In an off-list response, I pointed out that postfix start will do this automatically. Wietse $ man postfix ... check Warn about bad directory/file ownership or permissions, and cre- ate missing directories. start Start the Postfix mail system. This also runs the configuration check described above.
Re: Postfix queue data portabilty
Thank u guys, Mean taking proper care to the permissions and requeuing the file will sort the problem , this was just my imaginary condition that i was thought in worst case , does any one realy face and does the solution? Thanks Regards, Kshitij On Fri, Apr 13, 2012 at 11:20 AM, Stan Hoeppner s...@hardwarefreak.comwrote: On 4/12/2012 11:51 PM, kshitij mali wrote: my postfix queue data got increased to 850MB in size that is /data/postfix/queue directory all mail in this directory got stuck because the server hardware is down i had removed that hard drive and attached to another linux server and configured the postfix can any one tell me how to deliver this emails in the queue . $ postqueue -f -- Stan
Re: Postfix queue data portabilty
kshitij mali: Hi All, I am in condition of worst and had bad luck . my postfix queue data got increased to 850MB in size that is /data/postfix/queue directory all mail in this directory got stuck because the server hardware is down i had removed that hard drive and attached to another linux server and configured the postfix can any one tell me how to deliver this emails in the queue . Run postsuper repeatedly, until it stops saying that files have been renamed. Then, postfix start. Wietse
Postfix queue data portabilty
Hi All, I am in condition of worst and had bad luck . my postfix queue data got increased to 850MB in size that is /data/postfix/queue directory all mail in this directory got stuck because the server hardware is down i had removed that hard drive and attached to another linux server and configured the postfix can any one tell me how to deliver this emails in the queue . Regards, Kshitij Mali
Re: Postfix queue data portabilty
On 4/12/2012 11:51 PM, kshitij mali wrote: my postfix queue data got increased to 850MB in size that is /data/postfix/queue directory all mail in this directory got stuck because the server hardware is down i had removed that hard drive and attached to another linux server and configured the postfix can any one tell me how to deliver this emails in the queue . $ postqueue -f -- Stan
To find Return-Path from postfix queue
Hello, Is there any way, in which we could see Return-Path for a mail in postfix queue via any command or referring to any file. Using *postcat -q queueid* one can see the From address, sender:, but no Return-Path Also when is the Return-Path added into the postfix ? I was referring to one article ( http://www.linuxquestions.org/questions/linux-server-73/postfix-change-return-path-and-writing-own-script-for-master-cf-657934/) which states that The Return-Path is added by Postfix's cleanup agent just before final delivery. Is it true ? With Regards Anirudha Patil mobile no: 9221306620 email-id: anirudha.pa...@hotmail.com
Re: To find Return-Path from postfix queue
Am 23.03.2012 09:25, schrieb Anirudha Patil: Hello, Is there any way, in which we could see Return-Path for a mail in postfix queue via any command or referring to any file. Using *postcat -q queueid* one can see the From address, sender:, but no Return-Path the MTA is not interested in the From/To-Headers it woks always with envelopes signature.asc Description: OpenPGP digital signature
Re: To find Return-Path from postfix queue
Thank you Yes, i do know that, but i wanted to know if we can see the return-path for a mail ? Also any thoughts on if the Return-Path is added by postfix in header or its the same as the envelope sender. With Regards Anirudha Patil mobile no: 9221306620 email-id: anirudha.pa...@hotmail.com On Fri, Mar 23, 2012 at 1:58 PM, Reindl Harald h.rei...@thelounge.netwrote: Am 23.03.2012 09:25, schrieb Anirudha Patil: Hello, Is there any way, in which we could see Return-Path for a mail in postfix queue via any command or referring to any file. Using *postcat -q queueid* one can see the From address, sender:, but no Return-Path the MTA is not interested in the From/To-Headers it woks always with envelopes
Fwd: To find Return-Path from postfix queue
AND DO NOT REPLY MULTIPLE TIMES TO PREVENT OFFLIST- ANSWERS LIKE MINE BY REPLY TO THE WRONG DUPLICATE Original-Nachricht Betreff: Re: To find Return-Path from postfix queue Datum: Fri, 23 Mar 2012 09:39:59 +0100 Von: Reindl Harald h.rei...@thelounge.net Organisation: the lounge interactive design An: Anirudha Patil anirudha0...@gmail.com DO NOT reply with top-post after get a answer below! Am 23.03.2012 09:33, schrieb Anirudha Patil: With Regards Anirudha Patil mobile no: 9221306620 email-id: anirudha.pa...@hotmail.com mailto:anirudha.pa...@hotmail.com On Fri, Mar 23, 2012 at 1:58 PM, Reindl Harald h.rei...@thelounge.net mailto:h.rei...@thelounge.net wrote: Am 23.03.2012 09 tel:23.03.2012%2009:25, schrieb Anirudha Patil: Hello, Is there any way, in which we could see Return-Path for a mail in postfix queue via any command or referring to any file. Using *postcat -q queueid* one can see the From address, sender:, but no Return-Path the MTA is not interested in the From/To-Headers it woks always with envelopes Yes, i do know that, but i wanted to know if we can see the return-path for a mail ? Also any thoughts on if the Return-Path is added by postfix in header or its the same as the envelope sender what in works always with envelopes is unclear? http://en.wikipedia.org/wiki/Variable_envelope_return_path It works by using a different return path (also called envelope sender) -- Mit besten Grüßen, Reindl Harald the lounge interactive design GmbH A-1060 Vienna, Hofmühlgasse 17 CTO / software-development / cms-solutions p: +43 (1) 595 3999 33, m: +43 (676) 40 221 40 icq: 154546673, http://www.thelounge.net/ http://www.thelounge.net/signature.asc.what.htm signature.asc Description: OpenPGP digital signature
Re: To find Return-Path from postfix queue
In an older episode, on 2012-03-23 09:33, Anirudha Patil wrote: Also any thoughts on if the Return-Path is added by postfix in header or its the same as the envelope sender. The envelope sender is written into the mail as Return-Path by the MDAs (Mail Delivery Agents) when the mail is finally delivered. So yes, they are the same, but the Return-Path is only a reflection of the envelope sender. Hope this helps. wolfgang
Re: To find Return-Path from postfix queue
In an older episode, on 2012-03-23 09:57, Wolfgang Zeikat wrote: The envelope sender is written into the mail as Return-Path by the MDAs (Mail Delivery Agents) when the mail is finally delivered. So yes, they are the same, but the Return-Path is only a reflection of the envelope sender. See http://en.wikipedia.org/wiki/Return-Path
Re: To find Return-Path from postfix queue
Anirudha Patil: Hello, Is there any way, in which we could see Return-Path for a mail in postfix queue via any command or referring to any file. Use the mailq command. This shows the envelope sender (and still to be delivered recipients). Wietse
Re: To find Return-Path from postfix queue
On 23/3/2012 10:33 πμ, Anirudha Patil wrote: Also any thoughts on if the Return-Path is added by postfix in header or its the same as the envelope sender. See also: http://tech.groups.yahoo.com/group/postfix-users/message/283690 Nick
Re: moving entire postfix queue between machines
Em 15/09/11 19:16, mouss escreveu: in that scenario, is it OK to backup the /var/spool/postfix folder on the old machine and simply drop it on the new machine ? please stay with us. you said I do nt have a spare machine then ... on the nwe machine?? do you mean the new disk? sorry ... as 'new machine' i meant to say on the 'new OS installed on the new disk on the same machine'. Will that keep the messages that were on the queue ? Of course, i'll completly remove the var/spool/postfix on the new machine before doing that, im not thinking on merging queues in anyway, i just want to get the queue from the old machine and drop on the new one. will that work as expected, or it wont ? you need to to stop postfix first, then you can copy the queue. Nice to hear that ... i was already aware that the procedure would require postfix to be stopped, no problem on that. thanks for the advices mouss ! -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it
moving entire postfix queue between machines
Hi, I need to change the disks on my postfix machine and i do not have a spare machine, so i'll basically take out the old disk, put the new ones and reinstall the machine. in that scenario, is it OK to backup the /var/spool/postfix folder on the old machine and simply drop it on the new machine ? Will that keep the messages that were on the queue ? Of course, i'll completly remove the var/spool/postfix on the new machine before doing that, im not thinking on merging queues in anyway, i just want to get the queue from the old machine and drop on the new one. will that work as expected, or it wont ? thanks ! -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it
Re: moving entire postfix queue between machines
Le 15/09/2011 17:17, Leonardo Rodrigues a écrit : Hi, I need to change the disks on my postfix machine and i do not have a spare machine, so i'll basically take out the old disk, put the new ones and reinstall the machine. in that scenario, is it OK to backup the /var/spool/postfix folder on the old machine and simply drop it on the new machine ? please stay with us. you said I do nt have a spare machine then ... on the nwe machine?? do you mean the new disk? Will that keep the messages that were on the queue ? Of course, i'll completly remove the var/spool/postfix on the new machine before doing that, im not thinking on merging queues in anyway, i just want to get the queue from the old machine and drop on the new one. will that work as expected, or it wont ? you need to to stop postfix first, then you can copy the queue.
Inject email from web server to postfix queue
Hi List I don't know if this is possible But we have developed a website for a customer with a CRM backend, without around 72,000 subscribers. We want to update and manage the subscriber list on the webserver and generate the weekly mailshot, all these processes are running fine. Our solution was to create threads on the web server (running IIS) to generate emails to be sent by the middleware (a PHP like language). This does work fine up to around a couple of thousand emails, then it runs out of memory and the process dies. I am looking at some kind of throttle, but its not a very elegant solution. I thought if I could generate a postfix friendly file, I could create an SMB share and drop the 72,000 files into a folder that Postfix would see and then process. I have a live and very happy Postfix server, already managing the email for a couple of dozen domains. (Nothing very big, mostly 3/4 users on each). Which I don't want to risk messing up. Firstly, is the feasable? Can I create a new webqueue folder, drop the files in their and have postfix send them off? Can I copy a file from the mail queue as a template? Any thoughts on this appreciated. Regards Jon Harris
Re: Inject email from web server to postfix queue
On Mon, Sep 12, 2011 at 9:37 AM, Jon Harris j.har...@digital-ink.co.uk wrote: Hi List I don't know if this is possible It's not. Search the list archives, and there are plenty of people wanting an API for dropping mail straight into the postfix queue. I thought if I could generate a postfix friendly file, I could create an SMB share and drop the 72,000 files into a folder that Postfix would see and then process. Sure, you just need some middleware that understands SMTP and file locking. Write something up in perl, that will pickup new files on the system, lock them, pass them to postfix (either via sendmail or TCP/SMTP) and then remove the message file. Just be sure that your middleware application has some callback mechanism for registering success/fails when injecting into postfix. Postfix on its own (AFAIK) won't do this for you.
Re: Inject email from web server to postfix queue
On 12/09/2011 14:37, Jon Harris wrote: Hi List I don't know if this is possible But we have developed a website for a customer with a CRM backend, without around 72,000 subscribers. We want to update and manage the subscriber list on the webserver and generate the weekly mailshot, all these processes are running fine. Our solution was to create threads on the web server (running IIS) to generate emails to be sent by the middleware (a PHP like language). This does work fine up to around a couple of thousand emails, then it runs out of memory and the process dies. I am looking at some kind of throttle, but its not a very elegant solution. I thought if I could generate a postfix friendly file, I could create an SMB share and drop the 72,000 files into a folder that Postfix would see and then process. I may be missing something obvious here, but I don't see why a program that can't create 72,000 email messages without running out of memory will be able to create 72,000 files without failing for exactly the same reason. An email is, after all, merely a file which is written to an external process (an SMTP server, in this example example) rather than a physical location. In fact, I'm having difficulty imagining a reason why any competently written program can't send an arbitrarily large number of emails anyway - all it's doing is running round a loop repeatedly, and it doesn't matter whether it does it 72 times or 72,000 times. The real constraints are the capacity of your mail servers to handle that many emails. It sounds to me, therefore, is if what you've got is a serious flaw in your mail-generation program, and I'd be inclined to address that rather than trying to find a workaround for the problem that involves injecting mail directly into the queue. If you really can't fix the mail-generation program's sending limit (eg, because it's a third-party CMS that comes with the bug built-in), then, as has already been suggested, the simplest option is to use an intermediate program which takes files from your storage directory and then sends them out by email. That's not Postfix-specific; any program which can read a file and then write to SMTP will work with any MTA. It would be pretty trivial to code that in PHP or Perl, I'm presuming it would be equally simple in Python, Ruby, ASP or your programming language of choice. The only complex part of it is error handling to ensure that you don't send the same message twice to the same person or that you don't miss anyone out if any email fails to send. Mark -- Sent from my Babbage Difference Engine http://mark.goodge.co.uk http://www.ratemysupermarket.com
RE: Inject email from web server to postfix queue
Hi Mark I know this does sound odd, but the middleware uses its SQLite and its own SMTP queue management. Apparently when the queue gets bloated, things start to go wrong. Whereas creating 72,000 files on disk wouldn't be a problem. Yes, I agree this does expose a weekness in the middleware, but we are already committed to it. Most web applications only have to email an order or a contact form, etc. So I can't really blame the developers for me trying to use it as a mailing list manager. So, my idea was to use something a little more industrial strength to get the messages out. I was trying to draw a few things together. I know my Postfix server works beautifully (out of 18 servers, its my most reliable) and I know I can create a SMB share on it. I wanted to create a drop-folder. I understand from Peter Blair who also answered this thread that I should be able to write something in Perl to do this. Unfortunately, my Perl skills are pretty weak. Regards Jon Harris -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of Mark Goodge Sent: 12 September 2011 15:01 To: postfix-users@postfix.org Subject: Re: Inject email from web server to postfix queue On 12/09/2011 14:37, Jon Harris wrote: Hi List I don't know if this is possible But we have developed a website for a customer with a CRM backend, without around 72,000 subscribers. We want to update and manage the subscriber list on the webserver and generate the weekly mailshot, all these processes are running fine. Our solution was to create threads on the web server (running IIS) to generate emails to be sent by the middleware (a PHP like language). This does work fine up to around a couple of thousand emails, then it runs out of memory and the process dies. I am looking at some kind of throttle, but its not a very elegant solution. I thought if I could generate a postfix friendly file, I could create an SMB share and drop the 72,000 files into a folder that Postfix would see and then process. I may be missing something obvious here, but I don't see why a program that can't create 72,000 email messages without running out of memory will be able to create 72,000 files without failing for exactly the same reason. An email is, after all, merely a file which is written to an external process (an SMTP server, in this example example) rather than a physical location. In fact, I'm having difficulty imagining a reason why any competently written program can't send an arbitrarily large number of emails anyway - all it's doing is running round a loop repeatedly, and it doesn't matter whether it does it 72 times or 72,000 times. The real constraints are the capacity of your mail servers to handle that many emails. It sounds to me, therefore, is if what you've got is a serious flaw in your mail-generation program, and I'd be inclined to address that rather than trying to find a workaround for the problem that involves injecting mail directly into the queue. If you really can't fix the mail-generation program's sending limit (eg, because it's a third-party CMS that comes with the bug built-in), then, as has already been suggested, the simplest option is to use an intermediate program which takes files from your storage directory and then sends them out by email. That's not Postfix-specific; any program which can read a file and then write to SMTP will work with any MTA. It would be pretty trivial to code that in PHP or Perl, I'm presuming it would be equally simple in Python, Ruby, ASP or your programming language of choice. The only complex part of it is error handling to ensure that you don't send the same message twice to the same person or that you don't miss anyone out if any email fails to send. Mark -- Sent from my Babbage Difference Engine http://mark.goodge.co.uk http://www.ratemysupermarket.com
Re: Inject email from web server to postfix queue
On Mon, Sep 12, 2011 at 11:00 AM, Jon Harris j.har...@digital-ink.co.uk wrote: Hi Mark I know this does sound odd, but the middleware uses its SQLite and its own SMTP queue management. Bypass it. Tell it that the smart relay host is your postfix box. It is trying to do something it shouldn't. Postfix should be the only software that manages queues. Apparently when the queue gets bloated, things start to go wrong. Whereas creating 72,000 files on disk wouldn't be a problem. Yes, I agree this does expose a weekness in the middleware, but we are already committed to it. Most web applications only have to email an order or a contact form, etc. So I can't really blame the developers for me trying to use it as a mailing list manager. So, my idea was to use something a little more industrial strength to get the messages out. Yes, it is called postfix. On 12/09/2011 14:37, Jon Harris wrote: Hi List I don't know if this is possible But we have developed a website for a customer with a CRM backend, without around 72,000 subscribers. We want to update and manage the subscriber list on the webserver and generate the weekly mailshot, all these processes are running fine. Our solution was to create threads on the web server (running IIS) to generate emails to be sent by the middleware (a PHP like language). This does work fine up to around a couple of thousand emails, then it runs out of memory and the process dies. I am looking at some kind of throttle, but its not a very elegant solution. I thought if I could generate a postfix friendly file, I could create an SMB share and drop the 72,000 files into a folder that Postfix would see and then process. I may be missing something obvious here, but I don't see why a program that can't create 72,000 email messages without running out of memory will be able to create 72,000 files without failing for exactly the same reason. An email is, after all, merely a file which is written to an external process (an SMTP server, in this example example) rather than a physical location. In fact, I'm having difficulty imagining a reason why any competently written program can't send an arbitrarily large number of emails anyway - all it's doing is running round a loop repeatedly, and it doesn't matter whether it does it 72 times or 72,000 times. The real constraints are the capacity of your mail servers to handle that many emails. It sounds to me, therefore, is if what you've got is a serious flaw in your mail-generation program, and I'd be inclined to address that rather than trying to find a workaround for the problem that involves injecting mail directly into the queue. If you really can't fix the mail-generation program's sending limit (eg, because it's a third-party CMS that comes with the bug built-in), then, as has already been suggested, the simplest option is to use an intermediate program which takes files from your storage directory and then sends them out by email. That's not Postfix-specific; any program which can read a file and then write to SMTP will work with any MTA. It would be pretty trivial to code that in PHP or Perl, I'm presuming it would be equally simple in Python, Ruby, ASP or your programming language of choice. The only complex part of it is error handling to ensure that you don't send the same message twice to the same person or that you don't miss anyone out if any email fails to send. Mark -- Sent from my Babbage Difference Engine http://mark.goodge.co.uk http://www.ratemysupermarket.com
Re: Inject email from web server to postfix queue
Jon Harris: Hi List I don't know if this is possible But we have developed a website for a customer with a CRM backend, without around 72,000 subscribers. We want to update and manage the subscriber list on the webserver and generate the weekly mailshot, all these processes are running fine. Our solution was to create threads on the web server (running IIS) to generate emails to be sent by the middleware (a PHP like language). This does work fine up to around a couple of thousand emails, then it runs out of memory and the process dies. I am looking at some kind of throttle, but its not a very elegant solution. I thought if I could generate a postfix friendly file, I could create an SMB share and drop the 72,000 files into a folder that Postfix would see and then process. I have a live and very happy Postfix server, already managing the email for a couple of dozen domains. (Nothing very big, mostly 3/4 users on each). Which I don't want to risk messing up. Firstly, is the feasable? Can I create a new webqueue folder, drop the files in their and have postfix send them off? Can I copy a file from the mail queue as a template? Nope, you can't. The external interfaces are: SMTP protocol and the Postfix sendmail command. Wietse
Re: Write a mail directly to postfix queue
On 08/20/2011 12:27 AM, Stan Hoeppner wrote: On 8/19/2011 10:50 AM, Noel Jones wrote: But from the volume you've described, you'll have trouble without using specialized hardware. Or a small outbound relay farm comprised of, say, 4 relatively low end boxen, each with a low power dual core CPU, 8GB RAM, and a ~100GB SSD. Rewrite the app to submit via SMTP. The actual outbound is indeed a relay farm of outgoing mail servers behind LVS. The application server is just a generation box. And that is where the bottleneck is. An application change is anyway required, But what I would probably have to do is make parallel smtp connections (which will be much more complex to write :-) )
Write a mail directly to postfix queue
One customer of ours wants to send research reports to all his partners instantly after the data is available. Our custom application generates the mail files with the attachments. ( personalized per recipient ) These files are then read by an independent daemon and sent to postfix smtpd with multiple connections The biggest bottleneck to this system is the disk I/O. ( 15k rpm SAS drives ) To avoid the double write to disk can I write to postfix queue directly using some postfix library I dont want to make smtpd connections in the app because that slows down the app significantly , and also this is a serialized process. So sending mails serially slows down the general delivery Thanks Ram
Re: Write a mail directly to postfix queue
Am 19.08.2011 16:05, schrieb Ram: I dont want to make smtpd connections in the app because that slows down the app significantly and also this is a serialized process. So sending mails serially slows down the general delivery it is a bad design sending hughe bulk and normal mail-traffic with the same server/ip a) your slowing down problem b) reputation of this machine will be degraded sooner or later signature.asc Description: OpenPGP digital signature
Re: Write a mail directly to postfix queue
Ram: One customer of ours wants to send research reports to all his partners instantly after the data is available. Why not put the report on a website and send the partners email with a hyperlink? The biggest bottleneck to this system is the disk I/O. ( 15k rpm SAS drives ) Why not send ONE COPY of the report with multiple recipients? To avoid the double write to disk can I write to postfix queue directly using some postfix library Direct Postfix queue access is not and will never be supported. Wietse
Re: Write a mail directly to postfix queue
On 08/19/2011 07:59 PM, Wietse Venema wrote: Ram: One customer of ours wants to send research reports to all his partners instantly after the data is available. Why not put the report on a website and send the partners email with a hyperlink? The partners want them mailed .. just a (stupid ?) business requirement. The biggest bottleneck to this system is the disk I/O. ( 15k rpm SAS drives ) Why not send ONE COPY of the report with multiple recipients? These are personalized. Every partner gets his own data. To avoid the double write to disk can I write to postfix queue directly using some postfix library Direct Postfix queue access is not and will never be supported. I think I will have to make my app parallel processing. But is there a reason why you should not make a postfix-devel package :-) Postfix has a impressive set of features , this could be one of them too.
Re: Write a mail directly to postfix queue
On 08/19/2011 07:50 PM, Reindl Harald wrote: Am 19.08.2011 16:05, schrieb Ram: I dont want to make smtpd connections in the app because that slows down the app significantly and also this is a serialized process. So sending mails serially slows down the general delivery it is a bad design sending hughe bulk and normal mail-traffic with the same server/ip a) your slowing down problem b) reputation of this machine will be degraded sooner or later Why reputation? These are mails which partners pay to receive , not spam. Also the numbers are not too huge. It could be 50k-100k mails ..Only that they have to get sent ideally within 10 minutes .
Re: Write a mail directly to postfix queue
Am 19.08.2011 17:33, schrieb Ram: On 08/19/2011 07:50 PM, Reindl Harald wrote: Am 19.08.2011 16:05, schrieb Ram: I dont want to make smtpd connections in the app because that slows down the app significantly and also this is a serialized process. So sending mails serially slows down the general delivery it is a bad design sending hughe bulk and normal mail-traffic with the same server/ip a) your slowing down problem b) reputation of this machine will be degraded sooner or later Why reputation? These are mails which partners pay to receive , not spam tell this any automatism if a lot of customers are on big servers like gmail.com, yahoo.com. Also the numbers are not too huge. It could be 50k-100k mails this is more than enough you will not tell me that they are all valid forever what do you think happens if you try over and over submit mails from the same ip to the same host with invalid rcpt? sending this bulk with the same machine as normal musiness mails and thinking you have no increase in reputation and other problems is a little naive ..Only that they have to get sent ideally within 10 minutes this is simply impossible even if you get them to postfix, you will never ever get them out in this time and will be deferred on many destinations, if there is greylisting and you have a lot of mails to the same destination you will lose your reputation faster as you ever send signature.asc Description: OpenPGP digital signature
Re: Write a mail directly to postfix queue
On Fri, Aug 19, 2011 at 11:33 AM, Ram r...@netcore.co.in wrote: On 08/19/2011 07:50 PM, Reindl Harald wrote: Am 19.08.2011 16:05, schrieb Ram: I dont want to make smtpd connections in the app because that slows down the app significantly and also this is a serialized process. So sending mails serially slows down the general delivery it is a bad design sending hughe bulk and normal mail-traffic with the same server/ip a) your slowing down problem b) reputation of this machine will be degraded sooner or later Why reputation? These are mails which partners pay to receive , not spam. Also the numbers are not too huge. It could be 50k-100k mails ..Only that they have to get sent ideally within 10 minutes . $ units 2411 units, 71 prefixes, 33 nonlinear units You have: 10 seconds You want: 10 minutes * 166.7 / 0.006 Unless my quick math is wrong, that's 166 mail messages per second. I think that if you're worried about your harddrives not being up to snuff, you probably won't be sustaining these kinds of numbers. Especially if the message sizes are larger (ie, containing those base64 encoded attachments). Back to reputation, just because the recipient mailbox owner wants the mail, doesn't mean that the mailbox-owner's postmast will want the mail if you're bursting a lot of messages to multiple recipients under the same domain. Burstiness == spaminess in certain circles. If you're serious about this customer, consider placing them on a dedicated postfix instance, and if you're worried about IO latency, consider mounting the active queue as a tmpfs or ramdisk if you're system can support that VM-wise. But, that can be dangerous, since you will lose mail if your system goes down while a message is in a volatile storage mount. Good luck!
Re: Write a mail directly to postfix queue
On 8/19/2011 10:27 AM, Ram wrote: To avoid the double write to disk can I write to postfix queue directly using some postfix library Direct Postfix queue access is not and will never be supported. I think I will have to make my app parallel processing. But is there a reason why you should not make a postfix-devel package :-) Postfix has a impressive set of features , this could be one of them too. To avoid double writes, open an SMTP connection rather than a file. To get parallel delivery, use multiple SMTP connections. But from the volume you've described, you'll have trouble without using specialized hardware. -- Noel Jones
Re: Write a mail directly to postfix queue
On Fri, Aug 19, 2011 at 11:42:02AM -0400, Peter Blair wrote: On Fri, Aug 19, 2011 at 11:33 AM, Ram r...@netcore.co.in wrote: On 08/19/2011 07:50 PM, Reindl Harald wrote: Am 19.08.2011 16:05, schrieb Ram: I dont want to make smtpd connections in the app because that slows down the app significantly and also this is a serialized process. So sending mails serially slows down the general delivery it is a bad design sending hughe bulk and normal mail-traffic with the same server/ip a) your slowing down problem b) reputation of this machine will be degraded sooner or later Why reputation? These are mails which partners pay to receive , not spam. Also the numbers are not too huge. It could be 50k-100k mails ..Only that they have to get sent ideally within 10 minutes . $ units 2411 units, 71 prefixes, 33 nonlinear units You have: 10 seconds You want: 10 minutes * 166.7 / 0.006 Unless my quick math is wrong, that's 166 mail messages per second. I think that if you're worried about your harddrives not being up to snuff, you probably won't be sustaining these kinds of numbers. Especially if the message sizes are larger (ie, containing those base64 encoded attachments). Back to reputation, just because the recipient mailbox owner wants the mail, doesn't mean that the mailbox-owner's postmast will want the mail if you're bursting a lot of messages to multiple recipients under the same domain. Burstiness == spaminess in certain circles. If you're serious about this customer, consider placing them on a dedicated postfix instance, Yes, agreed. and if you're worried about IO latency, consider mounting the active queue as a tmpfs or ramdisk if you're system can support that VM-wise. But, that can be dangerous, since you will lose mail if your system goes down while a message is in a volatile storage mount. I wouldn't do it this way, for the very reason you gave. A safer option is to have the mail-generating software use the ramdisk. Presumably, that software could run again and regenerate any lost mails, and the Postfix logs would show exactly which ones were created and sent. -- Offlist mail to this address is discarded unless /dev/rob0 or not-spam is in Subject: header
contribution : little script to quickly analyze the postfix queue.
Hi,
This little script was developed which analyses output postqueue -p
and prints the report on the greatest abusers. It was developed
to quickly clear the queue of the deferred spam messages.
It is a stream processor and does not hogs memory , can analyze really
large volumes from postqueue -p.
Usage: postqueue -p | ./analyse_postfix_queue.pl
If is useful when used in conjunction to another script pfdel.pl
contributed by someone else , but repeated in this email.
Note: some paths in script pfdel.pl may need adjustment as per
local conditions.
# cat ./analyse_postfix_queue.pl
#!/usr/bin/perl -w
use strict;
my ($email,$total_rcpt) ;
my %h = ();
while () {
chomp ;
my $line = $_;
if($line =~ /^[A-Z0-9]{13}.*\s+(\S+)/)
{
$email = $1;
}
if($line =~ /^\s+(\S+)$/) {
$h{$email}++;
}
}
my @keys = sort {$h{$b} = $h{$a} } keys %h;
my $i=0;
printf(%3s %-30s %6s\n,SNO,E-mail,Total);
while($i 10)
{
$i++;
printf(%3d %-30s %6d\n,$i,$keys[$i-1],$h{$keys[$i-1]});
}
--
#!/usr/bin/perl -w
#
# pfdel - deletes message containing specified address from
# Postfix queue. Matches either sender or recipient address.
#
# Usage: pfdel email_address
#
use strict;
# Change these paths if necessary.
my $LISTQ = /opt/zimbra/postfix/sbin/postqueue -p;
my $POSTSUPER = /opt/zimbra/postfix/sbin/postsuper;
my $email_addr = ;
my $qid = ;
my $euid = $;
if ( @ARGV != 1 ) {
die Usage: pfdel email_address\n;
} else {
$email_addr = $ARGV[0];
}
if ( $euid != 0 ) {
die You must be root to delete queue files.\n;
}
open(QUEUE, $LISTQ |) ||
die Can't get pipe to $LISTQ: $!\n;
my $entry = QUEUE;# skip single header line
$/ = ;# Rest of queue entries print on
# multiple lines.
while ( $entry = QUEUE ) {
if ( $entry =~ / ($email_addr)$/m ) {
($qid) = split(/\s+/, $entry, 2);
$qid =~ s/[\*\!]//;
next unless ($qid);
#
# Execute postsuper -d with the queue id.
# postsuper provides feedback when it deletes
# messages. Let its output go through.
#
print deleting $qid ;
if ( system($POSTSUPER, -d, $qid) != 0 ) {
# If postsuper has a problem, bail.
die Error executing $POSTSUPER: error .
code . ($?/256) . \n;
}
print done\n;
}
}
close(QUEUE);
if (! $qid ) {
die No messages with the address $email_addr .
found in queue.\n;
}
exit 0;
Re: Postfix queue in Mysql ?
On 2010-12-29 10:14 AM, Joan Moreau wrote: But I have no car to fix . What is that story about ? In your first post, you vaguely described a 'problem': the postfix queue manager (qmgr) is taking far too much resources when the number of email pending is growing. Now, I did not rule out anything in any email. Yes you did: (dont tell me dbmail, I want to keep my dovecot imap/pop sever) You also said: (yes, a file system is made for storing files, but it is not at all made to execute queries on teh file tree (hey, it is a tree! not a rdbms ) which suggests that you feel a need to do lots of queries on the postfix queue - which suggests that you have [a] problem[s] that need to be fixed. Can you just tell me how to put the mailing queue in a DB (mysql database in my case) ? For the 5th or 6th time: IT IS NOT POSSIBLE. Now, if you would like some help with fixing the actual *problem[s]* (messages piling up in your queue[s], please follow the instructions you were given in the welcome message you received when you joined the list: TO REPORT A PROBLEM see: http://www.postfix.org/DEBUG_README.html#mail This usually means postfix version, output of postconf -n and unedited NON-verbose (unless verbose are specifically requested by someone helping you) logs exhibiting the problem. Other details, like contents of master.cf, and maybe even platform/OS details may be necessary for certain issues. -- Best regards, Charles
Re: Postfix queue in Mysql ?
On 2010-12-29 Joan Moreau wrote: Well, I am surprised by the tone of those emails. I, on the other hand, am very surprised, how you're consistently avoid giving this list any details about your configuration or your actual problem. Please either supply the information you've been asked for (several times), or stop wasting everyone's time. Regards Ansgar Wiechers -- Abstractions save us time working, but they don't save us time learning. --Joel Spolsky
Let's Kill This Thread (Was: Postfix queue in Mysql ?)
Le 29/12/2010 12:38, Ansgar Wiechers a écrit : On 2010-12-29 Joan Moreau wrote: Well, I am surprised by the tone of those emails. I, on the other hand, am very surprised, how you're consistently avoid giving this list any details about your configuration or your actual problem. Please either supply the information you've been asked for (several times), or stop wasting everyone's time. Let's please kill this thread. If OP provides more details about his real needs, it will be welcome.
Re: Postfix queue in Mysql ?
But I have no car to fix . What is that story about ? Now, I did not rule out anything in any email. Can you just tell me how to put the mailing queue in a DB (mysql database in my case) ? On Wed, 29 Dec 2010 18:04:45 +1100, James Gray wrote: On 29/12/2010, at 4:02 PM, Joan Moreau wrote: Well, I am surprised by the tone of those emails. Why? Do you tell you mechanic how to fix your car before he's even been informed what vehicle you drive? I am just asking if it exists a back-end that would replace the storage and management of the queue into mysql (i.e. put /var/spool/postfix into mysql tables). As you've bene told - no. Not unless you replace your backend with a dedicated DB system which you ruled out in your initial post. Cheers, James
Re: Postfix queue in Mysql ?
On 12/29/10 4:14 PM, Joan Moreau wrote: But I have no car to fix . What is that story about ? Now, I did not rule out anything in any email. Can you just tell me how to put the mailing queue in a DB (mysql database in my case) ? Exactly which part of NO YOU CAN NOT PUT THE QUEUE IN A DATABASE was hard to follow ? I'm sure it has been repeated several times now. I call troll. -- J.
Re: Postfix queue in Mysql ?
On Wed, 29 Dec 2010 10:14:49 -0500, Joan Moreau j...@grosjo.net wrote: Can you just tell me how to put the mailing queue in a DB (mysql database in my case) ? you may simply deploy MySQLfs (FUSE+MySQL) if you do not mind speeds, loads,...etc. http://sourceforge.net/projects/mysqlfs/ pgp7HNgIbhXrK.pgp Description: PGP signature
Re: Postfix queue in Mysql ?
On 12/29/2010 9:14 AM, Joan Moreau wrote: Can you just tell me how to put the mailing queue in a DB (mysql database in my case) ? Yes, rewrite postfix to use a DB as a backend rather than a file system. This will be a major redesign and not just a plugin. No one else is working on such a project, so you're on your own. If you require your mail queue to be stored in MySql, you'll need to find a different MTA because postfix just can't do that.
Postfix queue in Mysql ?
Hi, the postfix queue manager (qmgr) is taking far too much resources when the number of email pending is growing. Is there a wait to move /var/spool/postfix in a MySQL database ? (dont tell me dbmail, I want to keep my dovecot imap/pop sever) Thanks Joan
Re: Postfix queue in Mysql ?
On Tue, Dec 28, 2010 at 08:28:48AM -0500, Joan Moreau wrote: the postfix queue manager (qmgr) is taking far too much resources What does too much resources mean? CPU? disk I/O? RAM? when the number of email pending is growing. Treat the disease not the symptoms, why is the deferred queue large in the first place? Is there a way to move /var/spool/postfix in a MySQL database ? No, the Postfix queue uses the unix filesystem. Your mailstore can be a database if you have suitable delivery agents, but the Postfix queue cannot. This is a good thing. Now solve the real problem. -- Viktor.
Re: Postfix queue in Mysql ?
Joan Moreau: Hi, the postfix queue manager (qmgr) is taking far too much resources when the number of email pending is growing. Sorry, you are jumping to conclusions. There are many reasons why mail can pile up in the queue, and you have not given a shred of information that allows people here to help you. For more support, please see the mailing list welcome message, repeated below. Wietse TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html Thank you for using Postfix.
