Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[PGNet Dev]

On 10/28/20 2:38 PM, Wietse Venema wrote:

One possible way out is to skip the Postfix sendmail command, and
to use a "mini sendmail" program that submits mail via SMTP.


adding an msmtp sender as the VirusAction script in clamav milter, though a bit 
of 'extra', certainly is the simplest.

easy to implement, and seems to behave under my testing -- no more perms errors.

will watch for awhile.

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[PGNet Dev]

On 10/28/20 2:38 PM, Wietse Venema wrote:

One possible way out is to skip the Postfix sendmail command, and
to use a "mini sendmail" program that submits mail via SMTP.


i've typically got msmtp rattling around.


Obviously that will fail when Postfix is down.


noted.

not ideal, but not critical.


It is theoretically possible to build a 'submit via authenticated
SMTP' mode into the Postfix sendmail command, and/or to save
unsubmitted mail to ~/dead.letter, but both add complexity (new
bugs) and introduce new failure modes (more bugs).


i generally don't like the idea of adding more failure points.

i'm working to remove as much cruft as possible.  adding the notification 
script already is counterproductive to that goal.

seems to me there should be a way to give that extra script -- called by a service 
milter that _itself_ seems to have all its needed perms for its 'normal' ops -- the 
correct rights & perms to talk to sendmail.
from _outside_ of postfix.

which seems like not-a-postfix matter.

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[Wietse Venema]

PGNet Dev:
> my clamav-milter.conf includes
> 
>   VirusAction /usr/local/etc/clamav/scripts/virus-alert.sh
> 
> where that script _does_ invoke sendmail.
> 
> found this process
> 
>   ps ax | grep virus
>   15670 ?S  0:00 /bin/bash 
> /usr/local/etc/clamav/scripts/virus-alert.sh 
> Sanesecurity.Jurlbl.Auto.85a586.UNOFFICIAL 4DLaEC42Q6z2Q 
> ngocquyen...@gmail.com myu...@example.com Engineer Tee Shirt & Science Tee 
> Shirt  
> Tue, 27 Oct 2020 17:29:06 -0700
> 
> hanging around, obviously dealing with a spammy "Engineer Tee Shirt & Science 
> Tee Shirt" send.
> 
> i killed it & verified that the current stream of
> 
>   warning: mail_queue_enter: create file maildrop/731085.15673: 
> Permission denied
> 
> in postfix logs indeed stops.
> 
> so, seems like a likely bet!

One possible way out is to skip the Postfix sendmail command, and
to use a "mini sendmail" program that submits mail via SMTP.
Obviously that will fail when Postfix is down.

It is theoretically possible to build a 'submit via authenticated
SMTP' mode into the Postfix sendmail command, and/or to save
unsubmitted mail to ~/dead.letter, but both add complexity (new
bugs) and introduce new failure modes (more bugs).

Wietse

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[PGNet Dev]

On 10/28/20 11:36 AM, PGNet Dev wrote:

On 10/28/20 11:30 AM, Viktor Dukhovni wrote:

You might start with:

 # grep -r NoNewPrivileges /etc/systemd


i couldn't find any direct, relevant postdrop/maildrop, or NoNewPrivileges, 
references i chased sendmail usage instances instead.

i've clamav-milter in my milter chain on this box.

my clamav-milter.conf includes

VirusAction /usr/local/etc/clamav/scripts/virus-alert.sh

where that script _does_ invoke sendmail.

found this process

ps ax | grep virus
15670 ?S  0:00 /bin/bash 
/usr/local/etc/clamav/scripts/virus-alert.sh Sanesecurity.Jurlbl.Auto.85a586.UNOFFICIAL 
4DLaEC42Q6z2Q ngocquyen...@gmail.com myu...@example.com Engineer Tee Shirt & Science 
Tee Shirt  Tue, 
27 Oct 2020 17:29:06 -0700

hanging around, obviously dealing with a spammy "Engineer Tee Shirt & Science Tee 
Shirt" send.

i killed it & verified that the current stream of

warning: mail_queue_enter: create file maildrop/731085.15673: 
Permission denied

in postfix logs indeed stops.

so, seems like a likely bet!

clamav-milter does NOT have the

NoNewPermissions=true

set, and therefore defaults to

NoNewPermissions=false


That^ 'virus-alert.sh' -- exec'ing 'via' clamav-milter, obviously included in 
my postfix config -- needs to be given appropriate permissions for sending via 
sendmail.postfix.

The question is:

what/where is the correct method, so as NOT to break my postfix security 
model in some foolish manner ?

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[PGNet Dev]

On 10/28/20 11:30 AM, Viktor Dukhovni wrote:

You might start with:

 # grep -r NoNewPrivileges /etc/systemd

and all other directories with systemd unit files.


yup.  already done.

nothing --other than the now "=false" (need to double check if that's the same 
as _removing_ it ) in pflogsumm.service -- under /etc/systemd

there _are_ the standard instances in /usr/lib/systemd/system/systemd-*, but 
none that seem immediately relevant.

i know I've done this "to myself" with the install; i've just _never_ had the 
problem with any DIY-installed postfix*.

so, atm, elusive :-/

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[Viktor Dukhovni]

On Wed, Oct 28, 2020 at 11:22:55AM -0700, PGNet Dev wrote:
> On 10/28/20 10:32 AM, Viktor Dukhovni wrote:
> > Indeed a process with "no_new_privs" will not be able to run sendmail(1)
> > to submit new email.
> 
> noted.
> 
> that said, this _just_ reappeared here,
> 
>postfix/postdrop[15673]: warning: mail_queue_enter: create file 
> maildrop/678088.15673: Permission denied
> 
> so _not_ (just) pflogsumm :-/
> 
> ack that Postfix is just the msngr.
> 
> now to find what _other_ process is triggering this!

You might start with:

# grep -r NoNewPrivileges /etc/systemd

and all other directories with systemd unit files.

-- 
Viktor.

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[PGNet Dev]

On 10/28/20 10:32 AM, Viktor Dukhovni wrote:

Indeed a process with "no_new_privs" will not be able to run sendmail(1)
to submit new email.


noted.

that said, this _just_ reappeared here,

  postfix/postdrop[15673]: warning: mail_queue_enter: create file 
maildrop/678088.15673: Permission denied

so _not_ (just) pflogsumm :-/

ack that Postfix is just the msngr.

now to find what _other_ process is triggering this!

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[Viktor Dukhovni]

On Wed, Oct 28, 2020 at 06:19:10PM +0100, Bastian Blank wrote:

> > Barring interference from SELinux or AppArmour, ... this should not
> > happen unless file permissions change.
> 
> Maybe this was true ten years ago, but it is not longer.  The OP even
> mentioned something called "no new privileges", which is described only
> one tiny internet search away at
> https://www.kernel.org/doc/html/latest/userspace-api/no_new_privs.html.

The "..." after "SELinux or AppArmour" is inclusive of other controls
that modify expected POSIX semantics.

Indeed a process with "no_new_privs" will not be able to run sendmail(1)
to submit new email.

> It seems that you have to learn a lot about the security controls that
> a modern Linux provides.

A less patronising tone would be welcome...

-- 
Viktor.

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[Bastian Blank]

Hi Viktor

On Wed, Oct 28, 2020 at 01:00:35PM -0400, Viktor Dukhovni wrote:
> On Wed, Oct 28, 2020 at 09:01:38AM -0700, PGNet Dev wrote:
> > Oct 28 15:02:40 svr019 postfix/postdrop[64624]: warning: 
> > mail_queue_enter: create file maildrop/553726.64624: Permission denied
> > Oct 28 15:02:45 svr019 postfix/postdrop[32688]: warning: 
> > mail_queue_enter: create file maildrop/766615.32688: Permission denied
> Barring interference from SELinux or AppArmour, ... this should not
> happen unless file permissions change.

Maybe this was true ten years ago, but it is not longer.  The OP even
mentioned something called "no new privileges", which is described only
one tiny internet search away at
https://www.kernel.org/doc/html/latest/userspace-api/no_new_privs.html.

And yes, this flag is exactly what can cause this: it disables
suid/sgid, so the maildrop process runs without it and of course can't
write into the maildrop directory.

> Correct file permissions are set
> via "postfix set-permissions" (at package install time) and should not
> change thereafter.

The OP even showed that the permissions are correct, but you did not
cite it.

> Of course if SELinux et. al, decide to intervene, then you have to
> fix the relevant settings.

It seems that you have to learn a lot about the security controls that
a modern Linux provides.

Regards,
Bastian

-- 
I object to intellect without discipline;  I object to power without
constructive purpose.
-- Spock, "The Squire of Gothos", stardate 2124.5

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[Viktor Dukhovni]

On Wed, Oct 28, 2020 at 10:13:23AM -0700, PGNet Dev wrote:

> > For reference, on my system:
> > 
> >  $ postconf setgid_group
> >  setgid_group = maildrop
> >  $ ls -ld /var/spool/postfix/maildrop
> >  drwx-wx---  2 postfix  maildrop  2 Oct 28 12:52 
> > /var/spool/postfix/maildrop
> >  $ ls -l /usr/local/sbin/postdrop
> >  -rwxr-sr-x  1 root  maildrop  41656 Oct 25 03:44 
> > /usr/local/sbin/postdrop
> 
> here (fwiw, this is Fedora32-packaged Postfix et al), minor diff
> 
>postconf setgid_group
> !!  setgid_group = postdrop

The specific group name is not important.  It just has to be the
same group for the executable and the directory.

>ls -ld /var/spool/postfix/maildrop
>  drwx-wx--- 2 postfix postdrop 4.0K Oct 28 08:45 
> /var/spool/postfix/maildrop/
> 
>ls -l `which postdrop`
>  -rwxr-sr-x 1 root postdrop 25K Aug 31 02:52 /usr/sbin/postdrop*

These look OK.

> as set by a relatively new install -- i.e., _not_ manually by me,
> 
>ls -ald / /var /var/spool /var/spool/postfix
> 
>  dr-xr-xr-x. 22 root root 4.0K Sep  6 10:59 //
>  drwxr-xr-x. 21 root root 4.0K Oct 27 20:36 /var/
>  drwxr-xr-x  14 root root 4.0K Oct 25 19:05 /var/spool/
>  drwxr-xr-x  16 root root 4.0K Aug 31 02:52 /var/spool/postfix/

That "x." suggests some extended POSIX ACLs on / and /var, but they're
likely OK.  You can double-check these.

Otherwise, you can check that no kernel settings disable setgid
execution.

> so far, on this install, 'everything works' ... except for the
> eventual/random appearance of those^ complaints.

You'll need to figure why the EPERM happens.  Postfix is just
the messenger.

-- 
Viktor.

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[PGNet Dev]

On 10/28/20 10:00 AM, Viktor Dukhovni wrote:

On Wed, Oct 28, 2020 at 09:01:38AM -0700, PGNet Dev wrote:


Oct 28 15:02:40 svr019 postfix/postdrop[64624]: warning: 
mail_queue_enter: create file maildrop/553726.64624: Permission denied
Oct 28 15:02:45 svr019 postfix/postdrop[32688]: warning: 
mail_queue_enter: create file maildrop/766615.32688: Permission denied


Barring interference from SELinux or AppArmour,


neither in-place/enabled here atm


... this should not
happen unless file permissions change.  Correct file permissions are set
via "postfix set-permissions" (at package install time)


yup.  and, also tried re-exec'ing here.  so far, no curative effect by itself.


and should not change thereafter.


that's what i've always seen to date.  and why i suspect it's something _else_ 
i've done that's monkeying with it.


For reference, on my system:

 $ postconf setgid_group
 setgid_group = maildrop
 $ ls -ld /var/spool/postfix/maildrop
 drwx-wx---  2 postfix  maildrop  2 Oct 28 12:52 /var/spool/postfix/maildrop
 $ ls -l /usr/local/sbin/postdrop
 -rwxr-sr-x  1 root  maildrop  41656 Oct 25 03:44 /usr/local/sbin/postdrop


here (fwiw, this is Fedora32-packaged Postfix et al), minor diff

  postconf setgid_group
!!  setgid_group = postdrop

  ls -ld /var/spool/postfix/maildrop
drwx-wx--- 2 postfix postdrop 4.0K Oct 28 08:45 /var/spool/postfix/maildrop/

  ls -l `which postdrop`
-rwxr-sr-x 1 root postdrop 25K Aug 31 02:52 /usr/sbin/postdrop*


Provided that setgid bit and group of the postdrop executable
allows it to write to /var/spool/postfix/maildrop, and all
parent directories have "x" for all users, all should work.



If /, /var, /var/spool, /var/spool/postfix are not world-accessible
(x bit for "other"), or if /var/spool/maildrop is not group-wx, or
postdrop is not setgid, or has the wrong group, then things don't
work.


as set by a relatively new install -- i.e., _not_ manually by me,



  ls -ald / /var /var/spool /var/spool/postfix

dr-xr-xr-x. 22 root root 4.0K Sep  6 10:59 //

drwxr-xr-x. 21 root root 4.0K Oct 27 20:36 /var/

drwxr-xr-x  14 root root 4.0K Oct 25 19:05 /var/spool/

drwxr-xr-x  16 root root 4.0K Aug 31 02:52 /var/spool/postfix/





so far, on this install, 'everything works' ... except for the eventual/random 
appearance of those^ complaints.

after changing that NoNewPerms bit in the pflogsumm.sh, I'm seeing no further 
errors.
now that I've clearly jinxed it ... i'll keep watch some more.

Re: postfix queue perms' control for pflogsumm reporting? avoiding "warning: mail_queue_enter: create file maildrop/...: Permission denied"

[Viktor Dukhovni]

On Wed, Oct 28, 2020 at 09:01:38AM -0700, PGNet Dev wrote:

>   Oct 28 15:02:40 svr019 postfix/postdrop[64624]: warning: 
> mail_queue_enter: create file maildrop/553726.64624: Permission denied
>   Oct 28 15:02:45 svr019 postfix/postdrop[32688]: warning: 
> mail_queue_enter: create file maildrop/766615.32688: Permission denied

Barring interference from SELinux or AppArmour, ... this should not
happen unless file permissions change.  Correct file permissions are set
via "postfix set-permissions" (at package install time) and should not
change thereafter.

For reference, on my system:

$ postconf setgid_group
setgid_group = maildrop
$ ls -ld /var/spool/postfix/maildrop
drwx-wx---  2 postfix  maildrop  2 Oct 28 12:52 /var/spool/postfix/maildrop
$ ls -l /usr/local/sbin/postdrop
-rwxr-sr-x  1 root  maildrop  41656 Oct 25 03:44 /usr/local/sbin/postdrop

Provided that setgid bit and group of the postdrop executable
allows it to write to /var/spool/postfix/maildrop, and all
parent directories have "x" for all users, all should work.

If /, /var, /var/spool, /var/spool/postfix are not world-accessible
(x bit for "other"), or if /var/spool/maildrop is not group-wx, or
postdrop is not setgid, or has the wrong group, then things don't
work.

Of course if SELinux et. al, decide to intervene, then you have to
fix the relevant settings.

-- 
Viktor.

Re: Postfix queue

[j.emerlik]

I am generally using postfix-logwatch for tracking log files.

However, I was thinking about something that is working in real time,
scripts can be run by cron in some period time.
Then the best solution could be write some syslog phraser or redirect
syslog to some
application eg. written in Java to interpret log file in realtime.

Maybe someone has a better idea?

Re: Postfix queue

[j.emerlik]

I am generalny using postfix-logwatch for tracking log files.

However, I was thinking about something that is working in real time,
scripts can be run by cron in some period time.
Then the best could be write some syslog phraser or redirect syslog to some
application eg. written in Java to interpret log file in realtime.


2018-02-12 17:59 GMT+01:00 Wietse Venema :

> j.emerlik:
> > I would like have policy service that will be able to write do data base
> > some information eg. when exactly  message was sent, message ID, DSN if
> > soemthing goes wrong. That means it should be working with Postfix queue.
>
> You can use "postqueue -j" to get a machine-readable queue listing
> with arrival time, and why mail is still in the queue.
>
> However for historical information of past deliveries you will have
> to rely on Postfix logs.
>
> Wiietse
>

Re: Postfix queue

[Wietse Venema]

j.emerlik:
> I would like have policy service that will be able to write do data base
> some information eg. when exactly  message was sent, message ID, DSN if
> soemthing goes wrong. That means it should be working with Postfix queue.

You can use "postqueue -j" to get a machine-readable queue listing
with arrival time, and why mail is still in the queue.

However for historical information of past deliveries you will have
to rely on Postfix logs.

Wiietse

Re: Postfix queue

[Wietse Venema]

Viktor Dukhovni:
> 
> 
> > On Feb 12, 2018, at 10:06 AM, j.emerlik  wrote:
> > 
> > It is possible to write some policy service that will be working with 
> > postfix queue ?
> 
> No.  That's a bad idea anyway.  To track message flow, parse the logs.

The closest that comes to this is a daemon that responds to
transport_maps queries. If you must do that, I suggest using a
socketmap table.

http://www.postfix.org/transport.5.html
http://www.postfix.org/socket_table.5.html

But, it will drain performance.

Wietse

Re: Postfix queue

[Ralph Seichter]

On 12.02.2018 16:44, j.emerlik wrote:

> I would like have database and there information : Message ID, Sent
> Date (or last date of send trying), DSN, number of send attempts,
> Mail_From, RCPT_TO.

That type of information should be extracted from the Postfix logs,
as existing tools like 'pflogsumm' do.

-Ralph

Re: Postfix queue

[j.emerlik]

I would like have database and there information : Message ID, Sent Date
(or last date of send trying), DSN, number of send attempts, Mail_From,
RCPT_TO.

It would be helpful to create statistics  or check exacly what happened
with the messages sent, eg, six months ago.



2018-02-12 16:22 GMT+01:00 Ralph Seichter :

> On 12.02.2018 16:06, j.emerlik wrote:
>
> > I would like have policy service that will be able to write do data
> > base some information eg. when exactly message was sent, message ID,
> > DSN if soemthing goes wrong. That means it should be working with
> > Postfix queue.
>
> That's not really a specific description of your needs or design goals,
> so you can expect answers to be similarly vague. ;-) Have you looked
> into postqueue(1) and/or showq(8) yet?
>
> -Ralph
>
>

Re: Postfix queue

[Viktor Dukhovni]

> On Feb 12, 2018, at 10:06 AM, j.emerlik  wrote:
> 
> It is possible to write some policy service that will be working with postfix 
> queue ?

No.  That's a bad idea anyway.  To track message flow, parse the logs.

-- 
Viktor.

RE: Postfix queue

[Fazzina, Angelo]

Hi, I would think you could write a script to do what you need ?

Here is one I use that is in Python.


[root@mta3 alf02013]# Summary


   Usage:  Summary -s -h {-|POSTFIX_LOG} [ POSTFIX_LOG .. ]

   Summarize postfix mail log.  Gzipped files are OK.

   Print one line for each delivered email, with these columns

  TIME_RECEIVED   TIME_SENT   ELAPSED QUEUEID  SOURCE_IP
   AUTHENTICATE_USER  FINAL_STATUS  FROM_ADDR  TO_ADDRS

   OPTIONS
 -h  Print column headers
 -s  Include email subject (if in Postfix log)




-ANGELO FAZZINA

ITS Service Manager:
Spam and Virus Prevention
Mass Mailing
G Suite/Gmail

ang...@uconn.edu
University of Connecticut,  ITS, SSG, Server Systems
860-486-9075

From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] 
On Behalf Of j.emerlik
Sent: Monday, February 12, 2018 10:07 AM
To: postfix-users@postfix.org
Subject: Postfix queue

Lately I wrote in python postfix policy service that can do something for me 
what I want.
Now I am thinking about next service butI don't know maybe it is not possible.
That is my question:
There is posssible write some service similar  to eg.

check_policy_service unix:private/policy-spf
It is possible to write some policy service that will be working with postfix 
queue ?

I would like have policy service that will be able to write do data base some 
information eg. when exactly  message was sent, message ID, DSN if soemthing 
goes wrong. That means it should be working with Postfix queue.
But policy services can be configured with smtpd_sender_restrictions and 
smtpd_recipient_restrictions.
It 's possible configure some policy service with postfix queue ?
Regards,
MattX

Re: Postfix queue

[Ralph Seichter]

On 12.02.2018 16:06, j.emerlik wrote:

> I would like have policy service that will be able to write do data
> base some information eg. when exactly message was sent, message ID,
> DSN if soemthing goes wrong. That means it should be working with
> Postfix queue.

That's not really a specific description of your needs or design goals,
so you can expect answers to be similarly vague. ;-) Have you looked
into postqueue(1) and/or showq(8) yet?

-Ralph

Re: Postfix Queue

[Viktor Dukhovni]

On Sat, Mar 12, 2016 at 01:14:17AM -0800, Robert Wolfe wrote:

> "I read that postifx will retry emails for 5 days before it will stop and
> email will be removed from the queue. That is too much.

https://tools.ietf.org/html/rfc5321#section-4.5.4.1

   Retries continue until the message is transmitted or the sender gives
   up; the give-up time generally needs to be at least 4-5 days.  It MAY
   be appropriate to set a shorter maximum number of retries for non-
   delivery notifications and equivalent error messages than for
   standard messages.  The parameters to the retry algorithm MUST be
   configurable.

This advice should be taken seriously.  However, in some environments
it is also important to notify the sender that their email did not
get through in a timely manner.  Therefore,  reducing that timeout to
2-3 days is not unreasonable, in combination with

http://www.postfix.org/postconf.5.html#delay_warning_time

# Warn sender of trouble after 2 hours, continue to retry for 2 days
#
delay_warning_time = 2h
maximal_queue_lifetime = 2d

> "I want to set email delivery to be retried only 3 times before it is
> removed from the queue permanently. And maximum time email should be in
> queue should be only 3 hours.

This is a bad idea, and if the OP's organization is spamming and
cares not for the delivery of their email, they can use some other
MTA.  People who ask for short queue lifetime generally have some
other problem they've failed to describe or address.  The best
thing is to ask them what their real problem is, though it is
possible they have OCD and want to tune the queue lifetime "just
because"...

Postfix will only give up after a failed delivery that occurs at
least maximal_queue_lifetime after the message is received.  If
the mail queue has a multi-day backlog, this could in principle be
multiple days after the configured maximal_queue_lifetime.

> How do I set or configure the maximum delivery retry times/duration for
> emails in queue."

The relevant parameters are

http://www.postfix.org/postconf.5.html#maximal_queue_lifetime
http://www.postfix.org/postconf.5.html#bounce_queue_lifetime
http://www.postfix.org/postconf.5.html#delay_warning_time
http://www.postfix.org/postconf.5.html#maximal_backoff_time

For queue tuning advice, users should consult:

http://www.postfix.org/QSHAPE_README.html

-- 
Viktor.

Re: Postfix: queue size and replies

[Wietse Venema]

Peter:
> Hi lads (and girls), 
> 
> Is there an easy way to set the auto-replies (or answers, not sure how
> are they called) in postfix like greylist does "come back later", or
> "try the next available MX gateway" when my queue size reaches a certain
> amount? It often happens that my primary MX servers are busy crunching
> emails meanwhile other servers are slacking off.

That would require a policy server (postfwd, policyd, ...).

Wietse

Re: Postfix queue data portabilty

[Wietse Venema]

kshitij mali:
> Hi All,
> 
> 
> I am in condition of worst and had bad luck .
> 
> my postfix queue data got increased to 850MB in size that is
> "/data/postfix/queue" directory
> all mail in this directory got stuck because the server hardware is down i
> had removed that hard drive and attached to another linux server and
> configured the postfix
> can any one tell me how to deliver this emails in the queue .

Run "postsuper" repeatedly, until it stops saying that files have
been renamed.

Then, "postfix start".

Wietse

Re: Postfix queue data portabilty

[kshitij mali]

Thank u guys,

Mean taking proper care to the permissions and requeuing the file will sort
the problem ,

this was just my imaginary condition that i was thought in worst case ,
does any one realy face and does the solution?

Thanks & Regards,

Kshitij

On Fri, Apr 13, 2012 at 11:20 AM, Stan Hoeppner wrote:

> On 4/12/2012 11:51 PM, kshitij mali wrote:
>
> > my postfix queue data got increased to 850MB in size that is
> > "/data/postfix/queue" directory
> > all mail in this directory got stuck because the server hardware is down
> i
> > had removed that hard drive and attached to another linux server and
> > configured the postfix
> > can any one tell me how to deliver this emails in the queue .
>
> $ postqueue -f
>
> --
> Stan
>

Re: Postfix queue data portabilty

[Stan Hoeppner]

On 4/12/2012 11:51 PM, kshitij mali wrote:

> my postfix queue data got increased to 850MB in size that is
> "/data/postfix/queue" directory
> all mail in this directory got stuck because the server hardware is down i
> had removed that hard drive and attached to another linux server and
> configured the postfix
> can any one tell me how to deliver this emails in the queue .

$ postqueue -f

-- 
Stan

Re: Postfix queue in Mysql ?

[Charles Marcus]

On 2010-12-29 10:14 AM, Joan Moreau wrote:
> But I have no " car to fix" . What is that story about ?

In your first post, you vaguely described a 'problem':


"the postfix queue manager (qmgr) is taking far too much resources when
the number of email pending is growing."

> Now, I did not rule out anything in any email.

Yes you did:

"(dont tell me "dbmail", I want to keep my dovecot imap/pop sever)"

You also said:

"(yes, a file system is made for storing files, but it is not at all
made to execute queries on teh file tree (hey, it is a tree! not a rdbms )"

which suggests that you feel a need to do lots of queries on the postfix
queue - which suggests that you have [a] problem[s] that need to be fixed.

> Can you just tell me how to put the mailing queue in a DB (mysql
> database in my case) ?

For the 5th or 6th time: IT IS NOT POSSIBLE.

Now, if you would like some help with fixing the actual *problem[s]*
(messages piling up in your queue[s], please follow the instructions you
were given in the welcome message you received when you joined the list:

TO REPORT A PROBLEM see:
http://www.postfix.org/DEBUG_README.html#mail

This usually means postfix version, output of postconf -n and unedited
NON-verbose (unless verbose are specifically requested by someone
helping you) logs exhibiting the problem. Other details, like contents
of master.cf, and maybe even platform/OS details may be necessary for
certain issues.

-- 

Best regards,

Charles

Re: Postfix queue in Mysql ?

[Noel Jones]

On 12/29/2010 9:14 AM, Joan Moreau wrote:

Can you just tell me how to put the mailing queue in a DB
(mysql database in my case) ?



Yes, rewrite postfix to use a DB as a backend rather than a 
file system.  This will be a major redesign and not just a 
plugin.  No one else is working on such a project, so you're 
on your own.


If you require your mail queue to be stored in MySql, you'll 
need to find a different MTA because postfix just can't do that.

Re: Postfix queue in Mysql ?

[Tomoyuki Murakami]

On Wed, 29 Dec 2010 10:14:49 -0500, Joan Moreau  wrote:

> Can you just tell me how to put
> the mailing queue in a DB (mysql database in my case) ?

you may simply deploy MySQLfs (FUSE+MySQL) if you do not mind
speeds, loads,...etc.
http://sourceforge.net/projects/mysqlfs/


pgp7HNgIbhXrK.pgp
Description: PGP signature

Re: Postfix queue in Mysql ?

[Jeroen Geilman]

On 12/29/10 4:14 PM, Joan Moreau wrote:


But I have no " car to fix" . What is that story about ?

Now, I did not rule out anything in any email.

Can you just tell me how to put the mailing queue in a DB (mysql 
database in my case) ?




Exactly which part of NO YOU CAN NOT PUT THE QUEUE IN A DATABASE was 
hard to follow ?


I'm sure it has been repeated several times now.

I call troll.


--
J.

Re: Postfix queue in Mysql ?

[Joan Moreau]

But I have no " car to fix" . What is that story about ? 

Now, I did
not rule out anything in any email. 

Can you just tell me how to put
the mailing queue in a DB (mysql database in my case) ? 

On Wed, 29 Dec
2010 18:04:45 +1100, James Gray wrote: 

> On 29/12/2010, at 4:02 PM,
Joan Moreau wrote:
> 
>> Well, I am surprised by the tone of those
emails.
> Why? Do you tell you mechanic how to fix your car before he's
even been informed what vehicle you drive? 
> 
>> I am just asking if it
exists a back-end that would replace the storage and management of the
queue into mysql (i.e. put /var/spool/postfix into mysql tables).
> As
you've bene told - no. Not unless you replace your backend with a
dedicated DB system which you ruled out in your initial post. Cheers,
James

Re: Postfix queue in Mysql ?

[Ansgar Wiechers]

On 2010-12-29 Joan Moreau wrote:
> Well, I am surprised by the tone of those emails. 

I, on the other hand, am very surprised, how you're consistently avoid
giving this list any details about your configuration or your actual
problem.

Please either supply the information you've been asked for (several
times), or stop wasting everyone's time.

Regards
Ansgar Wiechers
-- 
"Abstractions save us time working, but they don't save us time learning."
--Joel Spolsky

Re: Postfix queue in Mysql ?

[James Gray]

On 29/12/2010, at 4:02 PM, Joan Moreau wrote:

> Well, I am surprised by the tone of those emails.

Why? Do you tell you mechanic how to fix your car before he's even been 
informed what vehicle you drive?

>  I am just asking if it exists a back-end that would replace the storage and 
> management of the queue into mysql (i.e. put /var/spool/postfix into mysql 
> tables).

As you've bene told - no.  Not unless you replace your backend with a dedicated 
DB system which you ruled out in your initial post.

Cheers,

James

smime.p7s
Description: S/MIME cryptographic signature

Re: Postfix queue in Mysql ?

[Joan Moreau]

Well, I am surprised by the tone of those emails. 

I am just asking
if it exists a back-end that would replace the storage and management of
the queue into mysql (i.e. put /var/spool/postfix into mysql tables).


(yes, a file system is made for storing files, but it is not at all
made to execute queries on teh file tree (hey, it is a tree! not a rdbms
) 

On Wed, 29 Dec 2010 09:29:12 +1100, James Gray wrote: 

>> On Tue,
28 Dec 2010 12:28:57 -0500 (EST), Wietse Venema wrote: 
>> 
>>> Joan
Moreau: 
>>> 
 Well, more clearly, my question is : How can I plug
Mysql as a backend of postfix to handle the mailq ?
>>> Please state the
PROBLEM instead of the SOLUTION. Wietse
> On 29/12/2010, at 8:29 AM,
Joan Moreau wrote: 
> 
>> I am just looking for a MySQL bakcend to
replace the hard-disk storage of the postfix mailqueue. This is not a
problem, this is something I am looking for.
> (Top posting
fixed...quoting might not be - curse you Apple Mail!) So where exactly
will the MySQL backend live if not on disk?? If your qmgr is choking on
I/O, then throwing a DB at it wont help, in fact I can think of a number
of ways this will make an I/O-challenged systems WORSE. I guess you
could find some tool (or write one) that abstracts a MySQL database out
to a file system, heck, I've seen all manner of weird things abstracted
to file systems. At the end of the day, a file system is essentially an
extremely specialised DB...for storing, accessing and manipulating
files. So far the list has asked for specifics about the problem. You
keep demanding a specific solution. Without knowing anything about your
setup, what testing you've done, heck, even the version of Postfix
you're on, how do you expect us to help you? Without being rude, have a
read of http://www.catb.org/~esr/faqs/smart-questions.html [1] -
specifically about half way down titled: "Be precise and informative
about your problem". Good luck, James


Links:
--
[1]
http://www.catb.org/~esr/faqs/smart-questions.html

Re: Postfix queue in Mysql ?

[Stan Hoeppner]

Joan Moreau put forth on 12/28/2010 3:29 PM:
> 
> 
> Well, no need to get angry.

No one is angry.  You're misreading "tone" as you're not a regular
member of this list.  Replace "angry" with "direct" and you've got the
correct tone.

> I am just looking for a MySQL bakcend to replace the hard-disk storage of
> the postfix mailqueue. This is not a problem, this is something I am
> looking for.

You are telling us that you want the queue somewhere else than on the
local disk in the Postfix server chassis.  You cannot use a remote
database for this, as you've been told already.  The queue is filesystem
based, and always will be.

You have a couple of options for remote storage:

1.  NFS mounted filesystem
Can be inexpensive, usually lower performance w/low price solution
You (one with knowledge) can easily build an NFS server, esp w/Linux
Staring price < $500 USD if "home built"

2.  SAN LUN via iSCSI, fiber channel, infiniband
Local filesystem, acts like "local" SCSI disk
More expensive than NFS, usually higher performance
More difficult to setup, staring price ~$8k USD

My guess is, in absence of further details, that you should simply
purchase another disk, install it in your Postfix server chassis, and
move the mail spool directory to the new disk.

These recommendations are based on the limited amount of current detail
you've given us.  More accurate details will allow for a more accurate
or better recommendation, especially if you do indeed have a clogged
deferred queue.  In such a case more hardware will not solve the
problem.  In such a case, you need to find out why your messages aren't
being delivered.  If you have been marked as a spammer by various
places, this could be one explanation for your backed up queue.

Again, providing more details would allow us to actually assist you
instead of guessing.

-- 
Stan

Re: Postfix queue in Mysql ?

[James Gray]

> On Tue, 28 Dec 2010 12:28:57 -0500 (EST), Wietse Venema wrote:
> 
>> Joan Moreau:
>>> Well, more clearly, my question is : How can I plug Mysql as a backend
>>> of postfix to handle the mailq ?
>> 
>> Please state the PROBLEM instead of the SOLUTION. Wietse

On 29/12/2010, at 8:29 AM, Joan Moreau wrote:

> I am just looking for a MySQL bakcend to replace the hard-disk storage of
> the postfix mailqueue. This is not a problem, this is something I am
> looking for.

(Top posting fixed...quoting might not be - curse you Apple Mail!)

So where exactly will the MySQL backend live if not on disk??  If your qmgr is 
choking on I/O, then throwing a DB at it wont help, in fact I can think of a 
number of ways this will make an I/O-challenged systems WORSE.  I guess you 
could find some tool (or write one) that abstracts a MySQL database out to a 
file system, heck, I've seen all manner of weird things abstracted to file 
systems.  At the end of the day, a file system is essentially an extremely 
specialised DB...for storing, accessing and manipulating files.

So far the list has asked for specifics about the problem.  You keep demanding 
a specific solution.  Without knowing anything about your setup, what testing 
you've done, heck, even the version of Postfix you're on, how do you expect us 
to help you?  Without being rude, have a read of 
http://www.catb.org/~esr/faqs/smart-questions.html - specifically about half 
way down titled: "Be precise and informative about your problem".

Good luck,

James

smime.p7s
Description: S/MIME cryptographic signature

Re: Postfix queue in Mysql ?

[Victor Duchovni]

On Tue, Dec 28, 2010 at 04:29:34PM -0500, Joan Moreau wrote:

>
> I am just looking for a MySQL bakcend to replace the hard-disk storage of
> the postfix mailqueue. This is not a problem, this is something I am
> looking for.

This is surely a "means" and not an "end". What real purpose would
storing the queue in MySQL serve?

-- 
Viktor.

Re: Postfix queue in Mysql ?

[Magnus Bäck]

On Tuesday, December 28, 2010 at 22:29 CET,
 Joan Moreau  wrote:

> I am just looking for a MySQL bakcend to replace the hard-disk storage
> of the postfix mailqueue. This is not a problem, this is something I
> am looking for.

Judging by your initial message in this thread I'd say your problem is
that qmgr is requiring too much resources (of some kind), or at least
that's what you think is the problem. Let us help you address that
problem and figure out the root cause. This is done by answering our
questions.

Please do not top-post.

-- 
Magnus Bäck
mag...@dsek.lth.se

Re: Postfix queue in Mysql ?

[Joan Moreau]

Well, no need to get angry.

I am just looking for a MySQL bakcend to replace the hard-disk storage 
of

the postfix mailqueue. This is not a problem, this is something I am
looking for.

Best,

Joan

On Tue, 28 Dec 2010 12:28:57 -0500 (EST), Wietse Venema wrote:


Joan Moreau:
Well, more clearly, my question is : How can I plug Mysql as a 
backend

of postfix to handle the mailq ?


Please state the PROBLEM instead of the SOLUTION. Wietse

Re: Postfix queue in Mysql ?

[Victor Duchovni]

On Tue, Dec 28, 2010 at 12:19:23PM -0500, Joan Moreau wrote:

> Well, more clearly, my question is : 
> 
> How can I plug Mysql as a
> backend of postfix to handle the mailq ? 

It was clear enough before. The answer is that this is not
possible. The queue is file-based by design.

The queue design is performant for deferred queues up to a few hundred
thousand messages. Beyond that, indeed you will have problems, but then
the real question is why does your deferred queue have 100k+ messages?

So if you want help, get help with keeping your incoming queue nearly
empty, your active queue at a few hundreds messages, and your deferred
queue at less than a few thousand messages.

-- 
Viktor.

Re: Postfix queue in Mysql ?

[Wietse Venema]

Joan Moreau:
> Well, more clearly, my question is : 
> 
> How can I plug Mysql as a
> backend of postfix to handle the mailq ? 

Please state the PROBLEM instead of the SOLUTION.

Wietse

> On Tue, 28 Dec 2010 12:00:04
> -0500 (EST), Wietse Venema wrote: 
> 
> > Joan Moreau:
> >> Hi, the postfix
> queue manager (qmgr) is taking far too much resources when the number of
> email pending is growing.
> > Sorry, you are jumping to conclusions. There
> are many reasons why mail can pile up in the queue, and you have not
> given a shred of information that allows people here to help you. For
> more support, please see the mailing list welcome message, repeated
> below. Wietse TO REPORT A PROBLEM see
> http://www.postfix.org/DEBUG_README.html#mail [1] TO (UN)SUBSCRIBE see
> http://www.postfix.org/lists.html [2] Thank you for using
> Postfix.
> 
> 
> Links:
> --
> [1]
> http://www.postfix.org/DEBUG_README.html#mail
> [2]
> http://www.postfix.org/lists.html

Re: Postfix queue in Mysql ?

[Joan Moreau]

Well, more clearly, my question is : 

How can I plug Mysql as a
backend of postfix to handle the mailq ? 

On Tue, 28 Dec 2010 12:00:04
-0500 (EST), Wietse Venema wrote: 

> Joan Moreau:
>> Hi, the postfix
queue manager (qmgr) is taking far too much resources when the number of
email pending is growing.
> Sorry, you are jumping to conclusions. There
are many reasons why mail can pile up in the queue, and you have not
given a shred of information that allows people here to help you. For
more support, please see the mailing list welcome message, repeated
below. Wietse TO REPORT A PROBLEM see
http://www.postfix.org/DEBUG_README.html#mail [1] TO (UN)SUBSCRIBE see
http://www.postfix.org/lists.html [2] Thank you for using
Postfix.


Links:
--
[1]
http://www.postfix.org/DEBUG_README.html#mail
[2]
http://www.postfix.org/lists.html

Re: Postfix queue in Mysql ?

[Wietse Venema]

Joan Moreau:
> Hi, 
> 
> the postfix queue manager (qmgr) is taking far too much
> resources when the number of email pending is growing. 

Sorry, you are jumping to conclusions.

There are many reasons why mail can pile up in the queue, and you
have not given a shred of information that allows people here to
help you.

For more support, please see the mailing list welcome message,
repeated below.

Wietse

TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail

TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html

Thank you for using Postfix.

Re: Postfix queue in Mysql ?

[Victor Duchovni]

On Tue, Dec 28, 2010 at 08:28:48AM -0500, Joan Moreau wrote:

> the postfix queue manager (qmgr) is taking far too much
> resources

What does "too much resources" mean? CPU? disk I/O? RAM?

> when the number of email pending is growing. 

Treat the disease not the symptoms, why is the deferred queue large in
the first place?

> Is there a way to move "/var/spool/postfix" in a MySQL database ? 

No, the Postfix queue uses the unix filesystem. Your mailstore can
be a database if you have suitable delivery agents, but the Postfix
queue cannot. This is a good thing. Now solve the real problem.

-- 
Viktor.

Re: postfix queue tuning

[Yaoxing]

We do have a web portal for users to s*bscribe or uns*bscribe
themselves. And each newsletter contains such a link in case users want
to uns*bscribe.
We already have a very big list which is filled with users who purchased
and s*bscribed in our web site. So there's no need to buy that from a
3rd party company. It's also the reason why we want to manage the list
ourselves. Handing such a list to a 3rd party ESP is always a potential
risk.
So far thank you for all your messages and merry christmas!

Regards,
Yaoxing


2010/12/25 5:48, Stan Hoeppner Wrote:

Yaoxing put forth on 12/24/2010 9:20 AM:


The list comes from our clients' subscriptions. However, we didn't
verify the ownership of the emails before which maybe lead to invalid
email addresses. This is what we can improve in future.

You should have already had a process in place for "list vetting" before
you decided to send bulk email on behalf of a "client".  You are
functioning as an Email Service Provider or "ESP" in your current
described capacity, or you are simply a spammer.  If you are bulk
emailing on behalf of a client _without_ already having a list vetting
process in place then you _ARE_ a spammer, by definition.

If you are a trying to be a "legit" bulk mailer, you need to implement
an automated COI process.  COI - Confirmed Opt In.  This means you have
a closed loop system for web based subscriptions where people can sign
up and remove themselves at will.  Typically this is done with a mailing
list manager program such as GNU Mailman or majordomo on *nix platforms.
  If you need to integrate this with a Microsoft database that's clue you
will have to create yourself or acquire software that's already been
written for this purpose.  There surely already is as you can't be the
first person to have such a need.

DONT'T USE THIRD PARTY LISTS!!  If your email subscribers actually want
your mailings, setup a subscription URL that they can visit to sign up
for your newsletters.

The fact that you have done none of these things already shows that you
are a spammer, whether you consider yourself one or not.  There are
masses of online resources that teach one to properly do legit bulk
mailing.  Google.

If you _are_ a legit bulk mailer, I'd suggest you sign up with an ESP
like Constant Contact and have them do your mailings for you.  They have
all of the vetting processes in place, and will teach you how to
properly do legit bulk mailing, COI, etc.  There obviously is a fee for
their services, but it will be more than cost effective if you have
legitimate subscribers who want your newsletters/product fliers/whatever.

Your current "shoot from the hip" method of bulk mailing is never going
to fly in today's world.  This is why you're already listed on dns black
lists and being wholesale rejected by Yahoo.

Given your current methods, you need to immediately stop all of your
bulk mailings until get a proper closed loop subscription service up and
running, whether you do that in house or with an ESP like Constant Contact.

If you are a legit bulk mailer, please have a Merry Christmas.  If you
are a spammer, choke on your eggnog.

Re: postfix queue tuning

[Stan Hoeppner]

Yaoxing put forth on 12/24/2010 9:20 AM:

> The list comes from our clients' subscriptions. However, we didn't
> verify the ownership of the emails before which maybe lead to invalid
> email addresses. This is what we can improve in future.

You should have already had a process in place for "list vetting" before
you decided to send bulk email on behalf of a "client".  You are
functioning as an Email Service Provider or "ESP" in your current
described capacity, or you are simply a spammer.  If you are bulk
emailing on behalf of a client _without_ already having a list vetting
process in place then you _ARE_ a spammer, by definition.

If you are a trying to be a "legit" bulk mailer, you need to implement
an automated COI process.  COI - Confirmed Opt In.  This means you have
a closed loop system for web based subscriptions where people can sign
up and remove themselves at will.  Typically this is done with a mailing
list manager program such as GNU Mailman or majordomo on *nix platforms.
 If you need to integrate this with a Microsoft database that's clue you
will have to create yourself or acquire software that's already been
written for this purpose.  There surely already is as you can't be the
first person to have such a need.

DONT'T USE THIRD PARTY LISTS!!  If your email subscribers actually want
your mailings, setup a subscription URL that they can visit to sign up
for your newsletters.

The fact that you have done none of these things already shows that you
are a spammer, whether you consider yourself one or not.  There are
masses of online resources that teach one to properly do legit bulk
mailing.  Google.

If you _are_ a legit bulk mailer, I'd suggest you sign up with an ESP
like Constant Contact and have them do your mailings for you.  They have
all of the vetting processes in place, and will teach you how to
properly do legit bulk mailing, COI, etc.  There obviously is a fee for
their services, but it will be more than cost effective if you have
legitimate subscribers who want your newsletters/product fliers/whatever.

Your current "shoot from the hip" method of bulk mailing is never going
to fly in today's world.  This is why you're already listed on dns black
lists and being wholesale rejected by Yahoo.

Given your current methods, you need to immediately stop all of your
bulk mailings until get a proper closed loop subscription service up and
running, whether you do that in house or with an ESP like Constant Contact.

If you are a legit bulk mailer, please have a Merry Christmas.  If you
are a spammer, choke on your eggnog.

-- 
Stan

Re: postfix queue tuning

[Victor Duchovni]

On Fri, Dec 24, 2010 at 11:38:19PM +0800, Yaoxing wrote:

> So I must scan the log for the list, isn't it? It works of course but is 
> there any more specific way to do that? because scanning spends a lot of 
> time, and you don't know where you stopped last time (or not easy to find 
> out). especially our front end platform is based on .NET which does not 
> work with Linux logs easily. It would be great if there's a way for postfix 
> to output such a list directly, so that other programs just read it 
> directly.

For bulk mail in high volume VERP is essentially a must, and software
needs to be developed and deployed to parse the bounces to determine which
are non-deliveries, and which are out-of-office or other auto-replies.

http://www.postfix.org/VERP_README.html

Just parsing logs is not enough, you need to parse returning bounce messages.

Reading the source code of bounce processors in Majordomo, Mailman, ...
will set you on the right path.

-- 
Viktor.

Re: postfix queue tuning

[pf at alt-ctrl-del.org]

"Wietse Venema":

Yaoxing:
True but there got to be some easy way to export that list, otherwise 
I'll have to delete the dead mails from our database manually from time 
to time. Any ideas how I can get everything work fluently? I mean, for 
example, every several days I get all dead mail addresses from postfix 
by maybe a windows console program, then delete them from our 
subscription database. But where can I get the list firstly? dead mails 
seems to have been deleted directly.


Postfix returns undeliverable mail to the envelope sender address.
Postfix logs the envelope sender address as:

   Dec 24 08:36:59 spike postfix/qmgr[31619]: 9E33D1F3EA7:
from=, size=X, nrcpt=Y (queue active)

The idea is to to feed the returned mail through a program that
extracts the failed recipient address and unsubscribes the recipient.

Wietse


If searching for 'status=bounced' is a valid way to find the addresses, then 
see the attached example.
#!/bin/bash
#
# cleanexample.sh
#
perl -ne 'print "DELETE FROM somedb.sometable WHERE somefield = \"$1\"\ LIMIT 1;\n" if 
/to=<(@.*)>.*status=bounced/' /var/log/mail.log >/root/cleanexample.sql
mysql -u someuser -psomepass -h 10.11.12.13 somedb < /root/cleanexample.sql

Re: postfix queue tuning

[Yaoxing]

So I must scan the log for the list, isn't it? It works of course but is 
there any more specific way to do that? because scanning spends a lot of 
time, and you don't know where you stopped last time (or not easy to 
find out). especially our front end platform is based on .NET which does 
not work with Linux logs easily. It would be great if there's a way for 
postfix to output such a list directly, so that other programs just read 
it directly.


Regards,
Yaoxing


2010/12/24 23:26, Wietse Venema Wrote:

Postfix returns undeliverable mail to the envelope sender address.
Postfix logs the envelope sender address as:

 Dec 24 08:36:59 spike postfix/qmgr[31619]: 9E33D1F3EA7:
from=, size=X, nrcpt=Y (queue active)

The idea is to to feed the returned mail through a program that
extracts the failed recipient address and unsubscribes the recipient.

Wietse

Re: postfix queue tuning

[Wietse Venema]

Yaoxing:
> True but there got to be some easy way to export that list, otherwise 
> I'll have to delete the dead mails from our database manually from time 
> to time. Any ideas how I can get everything work fluently? I mean, for 
> example, every several days I get all dead mail addresses from postfix 
> by maybe a windows console program, then delete them from our 
> subscription database. But where can I get the list firstly? dead mails 
> seems to have been deleted directly.

Postfix returns undeliverable mail to the envelope sender address.
Postfix logs the envelope sender address as:

Dec 24 08:36:59 spike postfix/qmgr[31619]: 9E33D1F3EA7:
from=, size=X, nrcpt=Y (queue active)

The idea is to to feed the returned mail through a program that
extracts the failed recipient address and unsubscribes the recipient.

Wietse

> Regards,
> Yaoxing
> 
> 
> 2010/12/24 20:21, Ralf Hildebrandt Wrote:
> > * Victor Duchovni:
> >
> >> - Remove non-working addresses promptly from your lists.
> > This step alon considerably improves reputation AND delivery time.
> >
> 
> 

Re: postfix queue tuning

[Yaoxing]

Thanks for the suggestions. Some comments below.

Regards,
Yaoxing


2010/12/24 9:17, Victor Duchovni Wrote:

On Fri, Dec 24, 2010 at 09:11:19AM +0800, ? wrote:
- Make sure your lists really contain users who want to receive the
   newsletter, not just users whose email address you happened to collect
   at some point in the past. Opt-in, not opt-out.
The list comes from our clients' subscriptions. However, we didn't 
verify the ownership of the emails before which maybe lead to invalid 
email addresses. This is what we can improve in future.

- Fix your DNS to have non-generic PTR records.
Does it matter if I use the same sub domain, which is "e", or should I 
use some other domain maybe "email" instead?

- Work with Yahoo, Barracuda, ... to be removed from blacklists, added
   to whitelists, ...

- Remove non-working addresses promptly from your lists.

- ...

Large scale bulk-email is 99% list management and 1% software/hardware
delivery infrastructure.

If you err on the side of not mailing users whose voluntary subscription
status is uncertain, you should be able to improve your sender
"reputation" over time.

Re: postfix queue tuning

[Yaoxing]

True but there got to be some easy way to export that list, otherwise 
I'll have to delete the dead mails from our database manually from time 
to time. Any ideas how I can get everything work fluently? I mean, for 
example, every several days I get all dead mail addresses from postfix 
by maybe a windows console program, then delete them from our 
subscription database. But where can I get the list firstly? dead mails 
seems to have been deleted directly.


Regards,
Yaoxing


2010/12/24 20:21, Ralf Hildebrandt Wrote:

* Victor Duchovni:


- Remove non-working addresses promptly from your lists.

This step alon considerably improves reputation AND delivery time.

Re: postfix queue tuning

[Ralf Hildebrandt]

* Victor Duchovni :

> - Remove non-working addresses promptly from your lists.

This step alon considerably improves reputation AND delivery time.

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

RE: postfix queue tuning

[Ramprasad A Padmanabhan]

-Original Message-
From: Yaoxing
Sent:  23/12/2010 10:37:58 pm
To: r...@netcore.co.in
Cc: postfix-users@postfix.org
Subject:  Re: postfix queue tuning

> I think the bandwidths is OK. I have a 100Mb > ethernet but until now it's 
> like15Mb/s according to
> iftop -i eth1
> For the concurrency issue, what parameter 
>  you suggest to change? I 

default_destination_concurrency_limit 
See how many smtp processes are already running and change.

Use with care if you are delivering to other servers not in your control. Not 
everyone would appreciate  too many simultaneous connections. You can also set 
different limits for different destination with trivial changes. Postfix is 
extremely configurable 

Re: postfix queue tuning

[Victor Duchovni]

On Fri, Dec 24, 2010 at 09:11:19AM +0800, ? wrote:

> I cannot really understand based on what do u insist on I'm spamer? Ok
> I didn't obfusecate the clients email that's my fault. But why do u
> think I post only a few lines at first?
> Besides, I did not exit from the thread, removing amavis does resolve
> my problem. And it's really too late for me if u ever noticed the
> timestamp of my mail, did u? What else? I'm still here reading ur
> posts but not hiding away after 5 hours sleep.
> If it's still not enough, what else would u suggest I can prove myself?

- Make sure your lists really contain users who want to receive the
  newsletter, not just users whose email address you happened to collect
  at some point in the past. Opt-in, not opt-out.

- Fix your DNS to have non-generic PTR records.

- Work with Yahoo, Barracuda, ... to be removed from blacklists, added
  to whitelists, ...

- Remove non-working addresses promptly from your lists.

- ...

Large scale bulk-email is 99% list management and 1% software/hardware
delivery infrastructure.

If you err on the side of not mailing users whose voluntary subscription
status is uncertain, you should be able to improve your sender
"reputation" over time.

-- 
Viktor.

Re: postfix queue tuning

[张耀星]

I cannot really understand based on what do u insist on I'm spamer? Ok
I didn't obfusecate the clients email that's my fault. But why do u
think I post only a few lines at first?
Besides, I did not exit from the thread, removing amavis does resolve
my problem. And it's really too late for me if u ever noticed the
timestamp of my mail, did u? What else? I'm still here reading ur
posts but not hiding away after 5 hours sleep.
If it's still not enough, what else would u suggest I can prove myself?

Sent from my iPhone

  2010-12-24，4:25，Stan Hoeppner  wrote：

> Ralf Hildebrandt put forth on 12/23/2010 1:45 PM:
>> * Wietse Venema :
>>
>>> I was getting suspicious because Yahoo is permanently refusing your
>>> mail, but this is bad:
>>>
>>>% host 195.151.228.67.b.barracudacentral.org
>>>195.151.228.67.b.barracudacentral.org has address 127.0.0.2
>>>
>>> BTW your SMTP server banner says e.dealextreme.com.
>>
>> It's also listed on SORBS as dealextreme.com
>
> Whether it's porn or gadgets, the abundant evidence gathered, starting
> with his own logs, makes it pretty clear that he's spamming.  If you
> buy
> his explanation of obfuscating his sender address, note that he didn't
> give a rip about obfuscating the recipient addresses, which will all
> undoubtedly now get scraped by other spammers from websites that
> archive
> this list.
>
> His hasty exit from the thread after being called out is simply the
> exclamation point.
>
> --
> Stan
>
>

Re: postfix queue tuning

[Stan Hoeppner]

Ralf Hildebrandt put forth on 12/23/2010 1:45 PM:
> * Wietse Venema :
> 
>> I was getting suspicious because Yahoo is permanently refusing your
>> mail, but this is bad:
>>
>> % host 195.151.228.67.b.barracudacentral.org
>> 195.151.228.67.b.barracudacentral.org has address 127.0.0.2
>>
>> BTW your SMTP server banner says e.dealextreme.com.
> 
> It's also listed on SORBS as dealextreme.com

Whether it's porn or gadgets, the abundant evidence gathered, starting
with his own logs, makes it pretty clear that he's spamming.  If you buy
his explanation of obfuscating his sender address, note that he didn't
give a rip about obfuscating the recipient addresses, which will all
undoubtedly now get scraped by other spammers from websites that archive
this list.

His hasty exit from the thread after being called out is simply the
exclamation point.

-- 
Stan

Re: postfix queue tuning

[Ralf Hildebrandt]

* Wietse Venema :

> I was getting suspicious because Yahoo is permanently refusing your
> mail, but this is bad:
> 
> % host 195.151.228.67.b.barracudacentral.org
> 195.151.228.67.b.barracudacentral.org has address 127.0.0.2
> 
> BTW your SMTP server banner says e.dealextreme.com.

It's also listed on SORBS as dealextreme.com

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: postfix queue tuning

[Ralf Hildebrandt]

* Victor Duchovni :

> It takes mail many days to get through the content filter. Fix your content
> filter.

Or circumvent it for this type of mail! If your KNOW what you're
sending out, why scan for viruses?

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: postfix queue tuning

[Yaoxing]

Although the subscription page is always open to our clients, we don't 
send newsletters until recently. That's why I suddenly get so many 
subscribers, valid and invalid, all at a time, and get so many troubles 
then.  Otherwise I won't work until 3:00AM.
Anyway, we're too far away from the topic. So far, I find that the issue 
is mainly caused by the content filter which is completely unnecessary. 
I removed the filter and tried

postsuper -r ALL
service postfix reload
It works great, now no congesting at all. Thank you so much. I'll follow 
your other suggestions to adjust our servers.
And for Stan, sorry to have made you misunderstand. Again, your figure 
150 tps disk IO helps to decide the image server should be removed from 
the mail server host.


Merry Christmas & Happy new year to all of you!

Regards,
Yaoxing


2010/12/24 3:11, Victor Duchovni Wrote:

On Fri, Dec 24, 2010 at 02:53:15AM +0800, Yaoxing wrote:


My company is a ecommerce company which send newsletters to our subscribed
clients weekly. we have nothing to do with spammers.

Sufficiently poor list management and/or privacy policies are
indistinguishable from spam. If you want to have your mail accepted widely
ensure that all email addresses are opted-in, never given away and never
purchased. Non-working addresses need to be reasonably promptly scrubbed
from the list.

 network:Class-Name:network
 network:ID:NETBLK-SOFTLAYER.67.228.128.0/19
 network:Auth-Area:67.228.128.0/19
 network:Network-Name:SOFTLAYER-67.228.128.0
 network:IP-Network:67.228.151.192/27
 network:IP-Network-Block:67.228.151.192-67.228.151.223
 network:Organization;I:DealExtreme
 network:Street-Address:PO BOX 779 TAI PO POST OFFICE
 network:City:TAI PO
 network:State:ot
 network:Postal-Code:Hong Kong
 network:Country-Code:HK
 network:Tech-Contact;I:sysadm...@softlayer.com
 network:Abuse-Contact;I:dealextr...@gmail.com
 network:Admin-Contact;I:IPADM258-ARIN
 network:Created:20100329
 network:Updated:20100329
 network:Updated-By:ipad...@softlayer.com

Also:

195.151.228.67.in-addr.arpa. IN PTR 67.228.151.195-static.reverse.softlayer.com.

is not a particularly wise PTR for a bulk-mail sending MTA.

Re: postfix queue tuning

[Yaoxing]

Well I don't want to make this thread look an advertisement. but as long 
as you found out already, try Alexa to get more about dealextreme.com 
which would prove to you we are not spammers.


Regards,
Yaoxing


2010/12/24 3:05, Wietse Venema Wrote:

Yaoxing:
I was getting suspicious because Yahoo is permanently refusing your
mail, but this is bad:

 % host 195.151.228.67.b.barracudacentral.org
 195.151.228.67.b.barracudacentral.org has address 127.0.0.2

BTW your SMTP server banner says e.dealextreme.com.

Wietse

Re: postfix queue tuning

[Victor Duchovni]

On Fri, Dec 24, 2010 at 02:53:15AM +0800, Yaoxing wrote:

> My company is a ecommerce company which send newsletters to our subscribed 
> clients weekly. we have nothing to do with spammers.

Sufficiently poor list management and/or privacy policies are
indistinguishable from spam. If you want to have your mail accepted widely
ensure that all email addresses are opted-in, never given away and never
purchased. Non-working addresses need to be reasonably promptly scrubbed
from the list.

network:Class-Name:network
network:ID:NETBLK-SOFTLAYER.67.228.128.0/19
network:Auth-Area:67.228.128.0/19
network:Network-Name:SOFTLAYER-67.228.128.0
network:IP-Network:67.228.151.192/27
network:IP-Network-Block:67.228.151.192-67.228.151.223
network:Organization;I:DealExtreme
network:Street-Address:PO BOX 779 TAI PO POST OFFICE
network:City:TAI PO
network:State:ot
network:Postal-Code:Hong Kong
network:Country-Code:HK
network:Tech-Contact;I:sysadm...@softlayer.com
network:Abuse-Contact;I:dealextr...@gmail.com
network:Admin-Contact;I:IPADM258-ARIN
network:Created:20100329
network:Updated:20100329
network:Updated-By:ipad...@softlayer.com

Also:

195.151.228.67.in-addr.arpa. IN PTR 67.228.151.195-static.reverse.softlayer.com.

is not a particularly wise PTR for a bulk-mail sending MTA.

-- 
Viktor.

Re: postfix queue tuning

[Wietse Venema]

Yaoxing:
> No this is a misunderstanding. I just masked my company's domain name 
> with xxx.com due to my policies. I used to mask it with abc.com but it 
> seems to be a TV chanel. I don't want to confuse people so I changed to 
> xxx.com, just randomly.
> My company is a ecommerce company which send newsletters to our 
> subscribed clients weekly. we have nothing to do with spammers.

I was getting suspicious because Yahoo is permanently refusing your
mail, but this is bad:

% host 195.151.228.67.b.barracudacentral.org
195.151.228.67.b.barracudacentral.org has address 127.0.0.2

BTW your SMTP server banner says e.dealextreme.com.

Wietse

Re: postfix queue tuning

[Yaoxing]

No this is a misunderstanding. I just masked my company's domain name 
with xxx.com due to my policies. I used to mask it with abc.com but it 
seems to be a TV chanel. I don't want to confuse people so I changed to 
xxx.com, just randomly.
My company is a ecommerce company which send newsletters to our 
subscribed clients weekly. we have nothing to do with spammers.


Regards,
Yaoxing


2010/12/24 2:43, Stan Hoeppner Wrote:

Yaoxing put forth on 12/23/2010 12:05 PM:
After doing some checking, it appears you are blasting out porn spam on
behalf of xxx.com whose domain registration is privacy protected by
contactprivacy.com in Toronto.  Thankfully you're sending less than what
you're attempting to send due to these technical problems.

According to your log snippet Yahoo has permanently deferred all your
mail from 67.228.151.195 (Softlayer IP space, a known spam haven ISP due
to a horribly vetted reseller program--actually no vetting at all) due
to rate violation.

The Barracuda BRBL dnsbl has your sending IP blacklisted:
http://www.mxtoolbox.com/SuperTool.aspx?action=blacklist%3a67.228.151.195

The IP resolved to your hostname in your sender address, e.xxx.com, is
listed with dnsbl Tiopan:
http://www.mxtoolbox.com/SuperTool.aspx?action=blacklist%3ae.xxx.com

and xxx.com has wildcard dns.  any.thing.i.type.xxx.com resolves to
66.114.124.140, which explains the above sender address resolved host IP
being on the Tiopan list.

All of these things are 100% spammer sign.  You appear to be affiliate
spamming (attempting to anyway) on behalf of xxx.com.

I'm sure other dnsbl listings are forthcoming, especially if we help you
get your queue problem resolved.  I'm thinking at this point it would be
best if we simply stopped assisting you, as we're not in the business of
helping spammers.

As you are a "green" spammer, demonstrated in spades here in this help
thread, I implore you to stop spamming now, and apply your (developing)
technical skill set to other, more socially positive, endeavors.

Re: postfix queue tuning

[Victor Duchovni]

On Fri, Dec 24, 2010 at 01:51:13AM +0800, Yaoxing wrote:

> Dec 23 11:38:35 e postfix/qmgr[29972]: 6FC51297C081: from=, 
> size=18380, nrcpt=1 (queue active)

Some new mail is entering the active queue either from "incoming" or
"deferred" queue.

Do you really want the hostname "e" in the envelope sender address?

> Dec 23 11:38:38 e postfix/smtp[34263]: 4E4C7297BE10: 
> to=, relay=127.0.0.1[127.0.0.1]:10024, conn_use=6, 
> delay=383792, delays=337025/46756/0/11, dsn=2.0.0, status=sent (250 2.0.0 
> Ok, id=33658-01-6, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 
> 50E79297B632)

Your content filter is FUBAR, processing 4-day old mail, it takes 11
seconds to process one message! This is absurdly high filter latency.

> Dec 23 11:38:38 e postfix/qmgr[29972]: 057DE2978A4D: from=, 
> size=45412, nrcpt=1 (queue active)
> Dec 23 11:38:40 e postfix/smtp[29728]: 057DE2978A4D: 
> to=, 
> relay=gmail-smtp-in.l.google.com[74.125.65.27]:25, delay=466500, 
> delays=466499/0.02/0.12/1.3, dsn=2.0.0, status=sent (250 2.0.0 OK 
> 1293125920 p26si24523792ybk.80)

Google accepts your mail in a timely manner, but it had been delayed
in "incoming" for a very long time...

> Dec 23 11:38:40 e postfix/smtp[29486]: 0809EFC9676: 
> to=, relay=mx1.hotmail.com[65.54.188.126]:25, 
> delay=318488, delays=318487/0.02/0.14/0.45, dsn=2.0.0, status=sent (250 
> <20101218080824.7201b2978...@e.xxx.com> Queued mail for delivery)

Hotmail also accepts your mail, be it with a bit more latency. Of
course given how congested your incoming queue is with pre-filter
mail, it takes forever to enter the active queue...

> Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host 
> g.mx.mail.yahoo.com[98.137.54.238] refused to talk to me: 421 4.7.1 [TS03] 
> All messages from 67.228.151.195 will be permanently deferred; Retrying 
> will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html

Yahoo wants nothing to do with you. And will NOT accept your mail,
no point in sending it unless you can resolve this with them. See
the friendly URL.

> Dec 23 11:38:42 e postfix/smtp[32893]: 9FE65FB01E1: 
> to=, relay=127.0.0.1[127.0.0.1]:10024, 
> conn_use=7, delay=454234, delays=407464/46764/0/6.2, dsn=2.0.0, status=sent 
> (250 2.0.0 Ok, id=33594-01-7, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: 
> queued as 70B572978A4D)

It is somewhat unlikely that  subscribed
to your newsletter. Such a role address is typically used for
"billing", not newsletter subscriptions. How do you manage subscriptions
to your list?

> Dec 23 11:38:51 e postfix/smtp[34263]: 7D28DFB50A0: 
> to=, relay=127.0.0.1[127.0.0.1]:10024, conn_use=7, 
> delay=383758, delays=336978/46767/0/13, dsn=2.0.0, status=sent (250 2.0.0 
> Ok, id=33658-01-7, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 
> 596FAFB405C)

More filter trouble. Your filter is FUBAR.

> Dec 23 11:39:55 e postfix/smtp[29728]: 7043F2979CE2: 
> to=, relay=mx2.comcast.net[76.96.30.116]:25, 
> delay=36569, delays=36567/0.03/0.87/1.8, dsn=5.1.1, status=bounced (host 
> mx2.comcast.net[76.96.30.116] said: 550 5.1.1 Not our Customer (in reply to 
> RCPT TO command))

Some of your list members don't exist. Do you remove non-working
addresses from your list?

-- 
Viktor.

Re: postfix queue tuning

[Stan Hoeppner]

Yaoxing put forth on 12/23/2010 12:05 PM:
> Then I think I didn't express it clearly. sorry for my bad English.
> I have like 400,000 subscribers. every week I send to all of them a news
> letter. Every 4 sec, I send out 1 mail to 1 person. I know it's very
> slow, but still it congests. That's why I'm wondering what's wrong. also
> the same server serves some images. So if it's because of the image
> server which takes most of the disk IO, I'll just remove the image
> server to another host.
> And about the filter, at the very beginning I don't know it causes so
> many troubles. that's what green hands do, right :)

After doing some checking, it appears you are blasting out porn spam on
behalf of xxx.com whose domain registration is privacy protected by
contactprivacy.com in Toronto.  Thankfully you're sending less than what
you're attempting to send due to these technical problems.

According to your log snippet Yahoo has permanently deferred all your
mail from 67.228.151.195 (Softlayer IP space, a known spam haven ISP due
to a horribly vetted reseller program--actually no vetting at all) due
to rate violation.

The Barracuda BRBL dnsbl has your sending IP blacklisted:
http://www.mxtoolbox.com/SuperTool.aspx?action=blacklist%3a67.228.151.195

The IP resolved to your hostname in your sender address, e.xxx.com, is
listed with dnsbl Tiopan:
http://www.mxtoolbox.com/SuperTool.aspx?action=blacklist%3ae.xxx.com

and xxx.com has wildcard dns.  any.thing.i.type.xxx.com resolves to
66.114.124.140, which explains the above sender address resolved host IP
being on the Tiopan list.

All of these things are 100% spammer sign.  You appear to be affiliate
spamming (attempting to anyway) on behalf of xxx.com.

I'm sure other dnsbl listings are forthcoming, especially if we help you
get your queue problem resolved.  I'm thinking at this point it would be
best if we simply stopped assisting you, as we're not in the business of
helping spammers.

As you are a "green" spammer, demonstrated in spades here in this help
thread, I implore you to stop spamming now, and apply your (developing)
technical skill set to other, more socially positive, endeavors.

-- 
Stan

Re: postfix queue tuning

[Yaoxing]

Then I think I didn't express it clearly. sorry for my bad English.
I have like 400,000 subscribers. every week I send to all of them a news 
letter. Every 4 sec, I send out 1 mail to 1 person. I know it's very 
slow, but still it congests. That's why I'm wondering what's wrong. also 
the same server serves some images. So if it's because of the image 
server which takes most of the disk IO, I'll just remove the image 
server to another host.
And about the filter, at the very beginning I don't know it causes so 
many troubles. that's what green hands do, right :)


Regards,
Yaoxing


2010/12/24 1:52, Stan Hoeppner Wrote:

Yaoxing put forth on 12/23/2010 11:29 AM:


relay=127.0.0.1[127.0.0.1]:10024

Why are you sending outbound newsletters through a content filter?  You
should already know that the content is not spam, and virus free, yes?
And if they are newsletters, why are you sending them every 4 seconds to
the entire subscriber base?

Speaking of that 4 second interval for each newsletter blast out, might
your application/content delivery be better served by an instant
messaging system or IRC channel?

One newsletter every 4 seconds = 21,600 newsletters per subscriber per
day.  No person would want that.  At the rate of one every 4 seconds,
this isn't a newsletter at all.

What, exactly, are you sending these subscribers every 4 seconds?  It is
probably a good time to explain what you're trying to accomplish with
your content delivery system.  Or am I completely misunderstanding the
"every 4 seconds" thing?

Re: postfix queue tuning

[Stan Hoeppner]

Yaoxing put forth on 12/23/2010 11:29 AM:

> relay=127.0.0.1[127.0.0.1]:10024

Why are you sending outbound newsletters through a content filter?  You
should already know that the content is not spam, and virus free, yes?
And if they are newsletters, why are you sending them every 4 seconds to
the entire subscriber base?

Speaking of that 4 second interval for each newsletter blast out, might
your application/content delivery be better served by an instant
messaging system or IRC channel?

One newsletter every 4 seconds = 21,600 newsletters per subscriber per
day.  No person would want that.  At the rate of one every 4 seconds,
this isn't a newsletter at all.

What, exactly, are you sending these subscribers every 4 seconds?  It is
probably a good time to explain what you're trying to accomplish with
your content delivery system.  Or am I completely misunderstanding the
"every 4 seconds" thing?

-- 
Stan

Re: postfix queue tuning

[Yaoxing]

Sorry, here's the full list with 100 lines:
Dec 23 11:38:35 e postfix/qmgr[29972]: 5E55BFC749F: removed
Dec 23 11:38:35 e postfix/qmgr[29972]: 6FC51297C081: 
from=, size=18380, nrcpt=1 (queue active)
Dec 23 11:38:38 e postfix/smtp[34263]: 4E4C7297BE10: 
to=, relay=127.0.0.1[127.0.0.1]:10024, conn_use=6, 
delay=383792, delays=337025/46756/0/11, dsn=2.0.0, status=sent (250 
2.0.0 Ok, id=33658-01-6, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: 
queued as 50E79297B632)

Dec 23 11:38:38 e postfix/qmgr[29972]: 4E4C7297BE10: removed
Dec 23 11:38:38 e postfix/qmgr[29972]: 057DE2978A4D: 
from=, size=45412, nrcpt=1 (queue active)
Dec 23 11:38:40 e postfix/smtp[29728]: 057DE2978A4D: 
to=, 
relay=gmail-smtp-in.l.google.com[74.125.65.27]:25, delay=466500, 
delays=466499/0.02/0.12/1.3, dsn=2.0.0, status=sent (250 2.0.0 OK 
1293125920 p26si24523792ybk.80)

Dec 23 11:38:40 e postfix/qmgr[29972]: 057DE2978A4D: removed
Dec 23 11:38:40 e postfix/qmgr[29972]: 0809EFC9676: 
from=, size=45409, nrcpt=1 (queue active)
Dec 23 11:38:40 e postfix/smtp[29486]: 0809EFC9676: 
to=, relay=mx1.hotmail.com[65.54.188.126]:25, 
delay=318488, delays=318487/0.02/0.14/0.45, dsn=2.0.0, status=sent (250 
<20101218080824.7201b2978...@e.xxx.com> Queued mail for delivery)

Dec 23 11:38:40 e postfix/qmgr[29972]: 0809EFC9676: removed
Dec 23 11:38:40 e postfix/qmgr[29972]: BABE3FB5C82: 
from=, size=45423, nrcpt=1 (queue active)
Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host 
g.mx.mail.yahoo.com[98.137.54.238] refused to talk to me: 421 4.7.1 
[TS03] All messages from 67.228.151.195 will be permanently deferred; 
Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host 
f.mx.mail.yahoo.com[98.137.54.237] refused to talk to me: 421 4.7.1 
[TS03] All messages from 67.228.151.195 will be permanently deferred; 
Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host 
i.mx.mail.yahoo.com[74.6.140.64] refused to talk to me: 421 4.7.1 [TS03] 
All messages from 67.228.151.195 will be permanently deferred; Retrying 
will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Dec 23 11:38:41 e postfix/smtp[29766]: BABE3FB5C82: host 
j.mx.mail.yahoo.com[66.94.237.64] refused to talk to me: 421 4.7.1 
[TS03] All messages from 67.228.151.195 will be permanently deferred; 
Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
Dec 23 11:38:42 e postfix/smtp[32893]: 9FE65FB01E1: 
to=, relay=127.0.0.1[127.0.0.1]:10024, 
conn_use=7, delay=454234, delays=407464/46764/0/6.2, dsn=2.0.0, 
status=sent (250 2.0.0 Ok, id=33594-01-7, from MTA([127.0.0.1]:10025): 
250 2.0.0 Ok: queued as 70B572978A4D)

Dec 23 11:38:42 e postfix/qmgr[29972]: 9FE65FB01E1: removed
Dec 23 11:38:42 e postfix/qmgr[29972]: E0934FB405C: 
from=, size=19318, nrcpt=1 (queue active)
Dec 23 11:38:42 e postfix/smtp[29766]: BABE3FB5C82: 
to=, 
relay=d.mx.mail.yahoo.com[209.191.88.254]:25, delay=177367, 
delays=177366/0.02/0.71/0, dsn=4.7.1, status=deferred (host 
d.mx.mail.yahoo.com[209.191.88.254] refused to talk to me: 421 4.7.1 
[TS03] All messages from 67.228.151.195 will be permanently deferred; 
Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html)
Dec 23 11:38:42 e postfix/qmgr[29972]: BABE3FB5C82: 
from=, status=expired, returned to sender

Dec 23 11:38:42 e postfix/qmgr[29972]: BABE3FB5C82: removed
Dec 23 11:38:42 e postfix/qmgr[29972]: 9E83FF3844C: 
from=, size=26887, nrcpt=1 (queue active)
Dec 23 11:38:45 e postfix/smtp[29728]: E0934FB405C: 
to=, 
relay=gmail-smtp-in.l.google.com[74.125.157.27]:25, delay=95227, 
delays=95223/0.06/0.1/3.3, dsn=2.0.0, status=sent (250 2.0.0 OK 
1293125925 72si16705236yhl.114)

Dec 23 11:38:45 e postfix/qmgr[29972]: E0934FB405C: removed
Dec 23 11:38:45 e postfix/qmgr[29972]: 64E45FBCD9D: 
from=, size=44493, nrcpt=1 (queue active)
Dec 23 11:38:51 e postfix/smtp[34263]: 7D28DFB50A0: 
to=, relay=127.0.0.1[127.0.0.1]:10024, 
conn_use=7, delay=383758, delays=336978/46767/0/13, dsn=2.0.0, 
status=sent (250 2.0.0 Ok, id=33658-01-7, from MTA([127.0.0.1]:10025): 
250 2.0.0 Ok: queued as 596FAFB405C)

Dec 23 11:38:51 e postfix/qmgr[29972]: 7D28DFB50A0: removed
Dec 23 11:38:51 e postfix/qmgr[29972]: CAFFA29EAC1A: 
from=, size=45398, nrcpt=1 (queue active)
Dec 23 11:38:54 e postfix/smtp[29486]: CAFFA29EAC1A: 
to=, relay=mxs.mail.ru[94.100.176.20]:25, delay=214118, 
delays=214116/0.02/0.51/1.8, dsn=2.0.0, status=sent (250 OK 
id=1PVp7t-0001oN-00)

Dec 23 11:38:54 e postfix/qmgr[29972]: CAFFA29EAC1A: removed
Dec 23 11:38:54 e postfix/qmgr[29972]: 87FB5F388D5: 
from=, size=26879, nrcpt=1 (queue active)
Dec 23 11:38:54 e postfix/smtp[32893]: 91EA6297F898: 
to=, relay=127.0.0.1[127.0.0.1]:10024, 
conn_use=8, delay=320559, delays=273776/46770/0/13, dsn=2.0.0, 
status=sent (250 2.0.0 Ok, id=33594-01-8, from MTA([127.0.0.1]:10025): 
250 2.0.0 Ok: queued as E4823FB50A0)

Dec 23 11:38:54 e postfi

Re: postfix queue tuning

[Victor Duchovni]

On Fri, Dec 24, 2010 at 01:37:48AM +0800, Yaoxing wrote:

> There's nothing in my hold queue. MailScanner do you mean amavis? I stopped 
> that 10 hours ago. but it doesn't seem to make the situation better.

You can't just "stop" the content filter, existing messages have the
content_filter transport recorded in the queue file, as evidenced by
your logs. To drop this record from existing messages you need to
run "postsuper -r ALL", which will move all mail from active, incoming
deferred (and I think also "hold") to "maildrop". From there, mail
will be re-processes as new mail in the queue with the new setting
of "content_filter" (as seen by pickup from main.cf or master.cf
overrides).

Restarting Postfix generally makes queue congestion worse.

> If I restart postfix, does it make the figure like that? because I noticed 
> mails in active queues went back to incoming queue while restarting.

Yes.

> So maybe those mails acts new mails?

No, they do not.

Do you have ANY logging of email going somewhere other than the content
filter?

So far all the evidence is that your content_filter is slow and/or busted,
there is nothing showing what happens post-filter. What happened to that
post-filter queue-id???

-- 
Viktor.

Re: postfix queue tuning

[Yaoxing]

There's nothing in my hold queue. MailScanner do you mean amavis? I 
stopped that 10 hours ago. but it doesn't seem to make the situation better.
If I restart postfix, does it make the figure like that? because I 
noticed mails in active queues went back to incoming queue while 
restarting. So maybe those mails acts new mails?


Regards,
Yaoxing


2010/12/24 1:29, Victor Duchovni Wrote:

On Fri, Dec 24, 2010 at 01:17:48AM +0800, Yaoxing wrote:


It's a newsletter group. because it's congesting so I stopped posting new
mails. I think that's why all mails are from 1280+ min ago.

No. This is wrong, the incoming queue contains fairly fresh mail.


I use
find active/ | wc -l
which gives me 20,002, while

OK, so the active queue has 20,000 single-recipient messages.


find incoming/ | wc -l
gives 130,000+
and the incoming queue is slowly decreasing.

Re: postfix queue tuning

[Victor Duchovni]

On Fri, Dec 24, 2010 at 01:29:00AM +0800, Yaoxing wrote:

>> Waste of time. Post NON-VERBOSE LOGGING by smtp(8) and qmgr(8).
>>
>>  logfiles=/some/where
>>  egrep 'postfix/(qmgr|smtp)\[' $logfiles | tail -100
>
> Dec 23 11:23:25 e postfix/qmgr[29972]: 3C15BFB9143: removed
> Dec 23 11:23:25 e postfix/qmgr[29972]: EB0E829E9CBC: from=, 
> size=18376, nrcpt=1 (queue active)
> Dec 23 11:23:27 e postfix/smtp[30439]: 88B5B2F385D0: 
> to=, relay=127.0.0.1[127.0.0.1]:10024, conn_use=7, 
> delay=410207, delays=364344/45849/0/13, dsn=2.0.0, status=sent (250 2.0.0 

It takes mail many days to get through the content filter. Fix your content
filter.

> Ok, id=30461-01-7, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 
> 722BA2988573)
> Dec 23 11:23:27 e postfix/qmgr[29972]: 88B5B2F385D0: removed
> Dec 23 11:23:27 e postfix/qmgr[29972]: CCB172D690D7: from=, 
> size=45424, nrcpt=1 (queue active)

This is not 100 log entries, how quickly do you rotate your logs? This
shows mail entering a disasterously slow content filter. Do you have a
filter loop? What happens to queue-id "722BA2988573"? Are you using both
amavisd-new and MailScanner?

>>  egrep ': warning: ' $logfiles | tail -20
> Dec 23 11:24:28 e postfix/qmgr[29972]: warning: mail for [127.0.0.1]:10024 
> is using up 2 of 2 active queue entries

Well, your content-filter is completely saturated to the point of being
non-functional..

-- 
Viktor.

Re: postfix queue tuning

[Victor Duchovni]

On Fri, Dec 24, 2010 at 01:17:48AM +0800, Yaoxing wrote:

> It's a newsletter group. because it's congesting so I stopped posting new 
> mails. I think that's why all mails are from 1280+ min ago.

No. This is wrong, the incoming queue contains fairly fresh mail.

> I use
> find active/ | wc -l
> which gives me 20,002, while

OK, so the active queue has 20,000 single-recipient messages.

> find incoming/ | wc -l
> gives 130,000+
> and the incoming queue is slowly decreasing.

Then fresh mail should be showing up in the active queue, but your
qshape output reports none.  you mentioned the ever-broken MailScanner,
which manipulates the Postfix queue in unsupported ways, how many files
are in your "hold" queue? Disable MainScanner and see whether you still
have problems. MailScanner is not supported here...

Post NON-VERBOSE LOGGING by smtp(8) and qmgr(8).

logfiles=/some/where
egrep 'postfix/(qmgr|smtp)\[' $logfiles | tail -100

Also post any signs of trouble:

egrep ': (error|fatal|panic): ' $logfiles | tail -100

And potential trouble:

egrep ': warning: ' $logfiles | tail -20

-- 
Viktor.

Re: postfix queue tuning

[Yaoxing]

see comments below.

2010/12/24 1:16, Victor Duchovni Wrote:

On Fri, Dec 24, 2010 at 01:07:58AM +0800, Yaoxing wrote:
Waste of time. Post NON-VERBOSE LOGGING by smtp(8) and qmgr(8).

logfiles=/some/where
egrep 'postfix/(qmgr|smtp)\[' $logfiles | tail -100

Dec 23 11:23:25 e postfix/qmgr[29972]: 3C15BFB9143: removed
Dec 23 11:23:25 e postfix/qmgr[29972]: EB0E829E9CBC: 
from=, size=18376, nrcpt=1 (queue active)
Dec 23 11:23:27 e postfix/smtp[30439]: 88B5B2F385D0: 
to=, relay=127.0.0.1[127.0.0.1]:10024, conn_use=7, 
delay=410207, delays=364344/45849/0/13, dsn=2.0.0, status=sent (250 
2.0.0 Ok, id=30461-01-7, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: 
queued as 722BA2988573)

Dec 23 11:23:27 e postfix/qmgr[29972]: 88B5B2F385D0: removed
Dec 23 11:23:27 e postfix/qmgr[29972]: CCB172D690D7: 
from=, size=45424, nrcpt=1 (queue active)

Also post any signs of trouble:

egrep ': (error|fatal|panic): ' $logfiles | tail -100

nothing

And potential trouble:

egrep ': warning: ' $logfiles | tail -20
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: mail for 
[127.0.0.1]:10024 is using up 2 of 2 active queue entries
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: you may need to reduce 
smtp-amavis connect and helo timeouts
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: so that Postfix quickly 
skips unavailable hosts
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: you may need to increase 
the main.cf minimal_backoff_time and maximal_backoff_time
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: so that Postfix wastes 
less time on undeliverable mail
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: you may need to increase 
the master.cf smtp-amavis process limit
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: please avoid flushing 
the whole queue when you have
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: lots of deferred mail, 
that is bad for performance
Dec 23 11:24:28 e postfix/qmgr[29972]: warning: to turn off these 
warnings specify: qmgr_clog_warn_time = 0

Re: postfix queue tuning

[Yaoxing]

It's a newsletter group. because it's congesting so I stopped posting 
new mails. I think that's why all mails are from 1280+ min ago.

I use
find active/ | wc -l
which gives me 20,002, while
find incoming/ | wc -l
gives 130,000+
and the incoming queue is slowly decreasing.

Regards,
Yaoxing


2010/12/24 1:01, Victor Duchovni Wrote:

On Thu, Dec 23, 2010 at 07:03:45PM +0800, Yaoxing wrote:


qshape active

  T  5 10 20 40 80 160 320 640 1280 1280+
 TOTAL 1000  0  0  0  0  0   0   0   00  1000
 gmail.com  254  0  0  0  0  0   0   0   00   254
   hotmail.com  168  0  0  0  0  0   0   0   00   168
 yahoo.com   70  0  0  0  0  0   0   0   0070
   comcast.net   14  0  0  0  0  0   0   0   0014
   msn.com   12  0  0  0  0  0   0   0   0012
   aol.com   11  0  0  0  0  0   0   0   0011
 sbcglobal.net   10  0  0  0  0  0   0   0   0010
  yahoo.com.br9  0  0  0  0  0   0   0   00 9
 hotmail.co.uk9  0  0  0  0  0   0   0   00 9
web.de8  0  0  0  0  0   0   0   00 8
   mail.ru8  0  0  0  0  0   0   0   00 8
   free.fr8  0  0  0  0  0   0   0   00 8
  yahoo.es7  0  0  0  0  0   0   0   00 7
hotmail.fr7  0  0  0  0  0   0   0   00 7
gmx.de6  0  0  0  0  0   0   0   00 6
  juno.com6  0  0  0  0  0   0   0   00 6
   cox.net5  0  0  0  0  0   0   0   00 5
  live.com5  0  0  0  0  0   0   0   00 5
  sympatico.ca5  0  0  0  0  0   0   0   00 5

So it seems that the active queue contains *only* mail that is 1 day
or so old, NO mail (for the destinations above) has entered the active
queue in 1280 minutes.


and for qshape incoming:


No mail in the incoming queue that is older than 640 minutes, so it
would seem that the active queue is not being drained at all. Is the
queue-manager even running? Are any attempted deliveries logged?

Some logging of qmgr(8) and smtp(8) activity is essential at this
juncture. It is implausible that no mail has entered the active queue
at all, while the incoming queue is full, unless your queue manager is
not running at all, or consistently crashes.

Are any of the messages in the queue addressed to thousands of recipients?
Can you report the file counts in the active and incoming queues?

cd /var/spool/postfix/active
find . ! -name '?' -print | wc -l
cd ../incoming
find . ! -name '?' -print | wc -l


  T  5 10 20 40 80 160 320 640 1280 1280+
 TOTAL 1000  9  7 33 47 79 181 393 2510 0
 gmail.com  221  1  2  6  9 18  45 100  400 0
   hotmail.com  153  1  1  4  9 15  32  63  280 0
 e.dealextreme.com  152  1  0  4  4  2  17  35  890 0
 yahoo.com   88  0  0  3  7  3  14  37  240 0
   msn.com   12  0  0  0  0  0   1   6   50 0
  yahoo.com.br   12  0  0  1  2  1   1   6   10 0
   aol.com9  0  0  0  0  1   1   5   20 0
   mail.ru8  0  0  0  0  1   2   3   20 0
   free.fr7  0  0  0  0  2   1   4   00 0
   comcast.net5  0  0  0  0  0   2   2   10 0
web.de4  0  0  0  0  1   0   2   10 0
  yahoo.es4  0  0  0  1  0   1   2   00 0
hotmail.fr4  0  0  0  0  0   0   2   20 0
  terra.com.br4  0  0  0  0  1   2   1   00 0
  sympatico.ca4  0  0  0  0  1   0   2   10 0
  ntlworld.com4  0  0  0  0  1   0   3   00 0
 bellsouth.net4  0  0  0  0  0   2   1   10 0
gmx.de3  0  0  0  0  1   0   2   00 0
   ono.com3  0  0  0  0  0   0   3   00 0

Re: postfix queue tuning

[Victor Duchovni]

On Fri, Dec 24, 2010 at 01:07:58AM +0800, Yaoxing wrote:

> I think the bandwidths is OK. I have a 100Mb ethernet but until now it's 
> like15Mb/s according to
> iftop -i eth1
> For the concurrency issue, what parameter would you suggest to change? I 
> found some parameters from the documents but do not fully understand them 
> yet.

Waste of time. Post NON-VERBOSE LOGGING by smtp(8) and qmgr(8).

logfiles=/some/where
egrep 'postfix/(qmgr|smtp)\[' $logfiles | tail -100

Also post any signs of trouble:

egrep ': (error|fatal|panic): ' $logfiles | tail -100

And potential trouble:

egrep ': warning: ' $logfiles | tail -20

-- 
Viktor.

Re: postfix queue tuning

[Yaoxing]

I think the bandwidths is OK. I have a 100Mb ethernet but until now it's 
like15Mb/s according to

iftop -i eth1
For the concurrency issue, what parameter would you suggest to change? I 
found some parameters from the documents but do not fully understand 
them yet.


Regards,
Yaoxing


2010/12/24 0:54, Ramprasad A Padmanabhan Wrote:



-Original Message-
From: Yaoxing
Sent:  23/12/2010 4:23:51 pm
To: Ramprasad
Cc: postfix-users@postfix.org
Subject:  Re: postfix queue tuning

Hi Ram,
I do have some more spare memory, but I'm afraid it doesn't resolve my
problem.
Let's say, my active queue is filled with 20,000 mails, but mails are
not going out but remains in memory. In this case if I increase active
queue size, I just put more mails in memory, they still don't go out.
What do you think

You must either have a network choke , or you must be reaching the concurrency 
limit

See your bandwidth graph , is there a flat line?

There is no point biting more than what you can chew. I would suggest do not 
let your active queue fill up, postfix clears slower with a full queue. Send 
the mails slower if possible.

You would probably end up clearing quicker.

Re: postfix queue tuning

[Victor Duchovni]

On Thu, Dec 23, 2010 at 07:03:45PM +0800, Yaoxing wrote:

> qshape active
>
>  T  5 10 20 40 80 160 320 640 1280 1280+
> TOTAL 1000  0  0  0  0  0   0   0   00  1000
> gmail.com  254  0  0  0  0  0   0   0   00   254
>   hotmail.com  168  0  0  0  0  0   0   0   00   168
> yahoo.com   70  0  0  0  0  0   0   0   0070
>   comcast.net   14  0  0  0  0  0   0   0   0014
>   msn.com   12  0  0  0  0  0   0   0   0012
>   aol.com   11  0  0  0  0  0   0   0   0011
> sbcglobal.net   10  0  0  0  0  0   0   0   0010
>  yahoo.com.br9  0  0  0  0  0   0   0   00 9
> hotmail.co.uk9  0  0  0  0  0   0   0   00 9
>web.de8  0  0  0  0  0   0   0   00 8
>   mail.ru8  0  0  0  0  0   0   0   00 8
>   free.fr8  0  0  0  0  0   0   0   00 8
>  yahoo.es7  0  0  0  0  0   0   0   00 7
>hotmail.fr7  0  0  0  0  0   0   0   00 7
>gmx.de6  0  0  0  0  0   0   0   00 6
>  juno.com6  0  0  0  0  0   0   0   00 6
>   cox.net5  0  0  0  0  0   0   0   00 5
>  live.com5  0  0  0  0  0   0   0   00 5
>  sympatico.ca5  0  0  0  0  0   0   0   00 5

So it seems that the active queue contains *only* mail that is 1 day
or so old, NO mail (for the destinations above) has entered the active
queue in 1280 minutes.

> and for qshape incoming:
>

No mail in the incoming queue that is older than 640 minutes, so it
would seem that the active queue is not being drained at all. Is the
queue-manager even running? Are any attempted deliveries logged?

Some logging of qmgr(8) and smtp(8) activity is essential at this
juncture. It is implausible that no mail has entered the active queue
at all, while the incoming queue is full, unless your queue manager is
not running at all, or consistently crashes.

Are any of the messages in the queue addressed to thousands of recipients?
Can you report the file counts in the active and incoming queues?

cd /var/spool/postfix/active
find . ! -name '?' -print | wc -l
cd ../incoming
find . ! -name '?' -print | wc -l

>  T  5 10 20 40 80 160 320 640 1280 1280+
> TOTAL 1000  9  7 33 47 79 181 393 2510 0
> gmail.com  221  1  2  6  9 18  45 100  400 0
>   hotmail.com  153  1  1  4  9 15  32  63  280 0
> e.dealextreme.com  152  1  0  4  4  2  17  35  890 0
> yahoo.com   88  0  0  3  7  3  14  37  240 0
>   msn.com   12  0  0  0  0  0   1   6   50 0
>  yahoo.com.br   12  0  0  1  2  1   1   6   10 0
>   aol.com9  0  0  0  0  1   1   5   20 0
>   mail.ru8  0  0  0  0  1   2   3   20 0
>   free.fr7  0  0  0  0  2   1   4   00 0
>   comcast.net5  0  0  0  0  0   2   2   10 0
>web.de4  0  0  0  0  1   0   2   10 0
>  yahoo.es4  0  0  0  1  0   1   2   00 0
>hotmail.fr4  0  0  0  0  0   0   2   20 0
>  terra.com.br4  0  0  0  0  1   2   1   00 0
>  sympatico.ca4  0  0  0  0  1   0   2   10 0
>  ntlworld.com4  0  0  0  0  1   0   3   00 0
> bellsouth.net4  0  0  0  0  0   2   1   10 0
>gmx.de3  0  0  0  0  1   0   2   00 0
>   ono.com3  0  0  0  0  0   0   3   00 0

-- 
Viktor.

RE: postfix queue tuning

[Ramprasad A Padmanabhan]

-Original Message-
From: Yaoxing
Sent:  23/12/2010 4:23:51 pm
To: Ramprasad
Cc: postfix-users@postfix.org
Subject:  Re: postfix queue tuning

Hi Ram,
I do have some more spare memory, but I'm afraid it doesn't resolve my 
problem.
Let's say, my active queue is filled with 20,000 mails, but mails are 
not going out but remains in memory. In this case if I increase active 
queue size, I just put more mails in memory, they still don't go out. 
What do you think

You must either have a network choke , or you must be reaching the concurrency 
limit 

See your bandwidth graph , is there a flat line?

There is no point biting more than what you can chew. I would suggest do not 
let your active queue fill up, postfix clears slower with a full queue. Send 
the mails slower if possible.

You would probably end up clearing quicker. 

Re: postfix queue tuning

[Wietse Venema]

Yaoxing:
> @Stan
> It's OK, you helped a lot already. And your figure is a very good 
> reference for me. Thank you very much.
> 
> @Wietse
> I'll try not to use relay host to check the speed.
> And sorry if I didn't express myself clearly, I don't mean to complain 
> anything. Just don't know what's wrong and what to do because I'm a 
> green hand. I pasted the changed parameters below. removed some lines 
> because I think it's not related and don't want to make the list too 
> long. (it's still very long I know ;)
> 
> debug_peer_level = 2

If you are concerned about performance, then don't turn on verbose
logging. This alone can easily double the load on your disk, and
as we know the disk is often the slowest component in the system.

If your performance still is poor, keep removing main.cf changes
that not absolutely necessary (such as maximal/minimal backoff time
and so on).

If the performance is still poor, report "postconf -n" and qshape
results.

Wietse

Re: postfix queue tuning

[Yaoxing]

@Stan
It's OK, you helped a lot already. And your figure is a very good 
reference for me. Thank you very much.


@Wietse
I'll try not to use relay host to check the speed.
And sorry if I didn't express myself clearly, I don't mean to complain 
anything. Just don't know what's wrong and what to do because I'm a 
green hand. I pasted the changed parameters below. removed some lines 
because I think it's not related and don't want to make the list too 
long. (it's still very long I know ;)


debug_peer_level = 2
default_process_limit = 100
enable_original_recipient = no
inet_interfaces = all
inet_protocols = all
maximal_backoff_time = 4000s
maximal_queue_lifetime = 1d
message_size_limit = 15728640
minimal_backoff_time = 300s
mydestination = $myhostname, localhost, localhost.localdomain, 
localhost.$myhostname

mydomain = abc.com
myhostname = e.abc.com
mynetworks = 127.0.0.0/8, 10.12.157.130/32, 10.12.157.132/32, 
127.0.0.1/32, 10.9.76.200/32, 10.9.76.196/32, 10.9.76.194/32, 
10.16.117.68/32

mynetworks_style = subnet
myorigin = e.abc.com
newaliases_path = /usr/bin/newaliases.postfix
proxy_read_maps = $canonical_maps $lmtp_generic_maps 
$local_recipient_maps $mydestination $mynetworks $recipient_bcc_maps 
$recipient_canonical_maps $relay_domains $relay_recipient_maps 
$relocated_maps $sender_bcc_maps $sender_canonical_maps 
$smtp_generic_maps $smtpd_sender_login_maps $transport_maps 
$virtual_alias_domains $virtual_alias_maps $virtual_mailbox_domains 
$virtual_mailbox_maps

queue_run_delay = 300s
recipient_bcc_maps = 
proxy:mysql:/etc/postfix/mysql_recipient_bcc_maps_domain.cf, 
proxy:mysql:/etc/postfix/mysql_recipient_bcc_maps_user.cf
relay_domains = $mydestination, 
proxy:mysql:/etc/postfix/mysql_relay_domains.cf
relay_recipient_maps = 
proxy:mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
sender_bcc_maps = 
proxy:mysql:/etc/postfix/mysql_sender_bcc_maps_domain.cf, 
proxy:mysql:/etc/postfix/mysql_sender_bcc_maps_user.cf

smtp_connect_timeout = 15
smtp_helo_timeout = 15
smtpd_data_restrictions = reject_unauth_pipelining
smtpd_enforce_tls = no
smtpd_error_sleep_time = 0
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks,permit_sasl_authenticated, 
check_helo_access pcre:/etc/postfix/helo_access.pcre
smtpd_recipient_restrictions = reject_unknown_sender_domain, 
reject_unknown_recipient_domain, reject_non_fqdn_sender, 
reject_non_fqdn_recipient, reject_unlisted_recipient, permit_mynetworks, 
permit_sasl_authenticated, reject_unauth_destination, 
reject_non_fqdn_helo_hostname, reject_invalid_helo_hostname, 
check_policy_service inet:127.0.0.1:10031

smtpd_reject_unlisted_recipient = yes
smtpd_reject_unlisted_sender = yes
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = no
smtpd_sasl_local_domain =
smtpd_sasl_path = dovecot-auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_sender_login_maps = 
proxy:mysql:/etc/postfix/mysql_sender_login_maps.cf

smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated
smtpd_tls_loglevel = 0
smtpd_tls_security_level = may
tls_random_source = dev:/dev/urandom
transport_maps = proxy:mysql:/etc/postfix/mysql_transport_maps_user.cf, 
proxy:mysql:/etc/postfix/mysql_transport_maps_domain.cf

unknown_local_recipient_reject_code = 550
virtual_alias_maps = proxy:mysql:/etc/postfix/mysql_virtual_alias_maps.cf
virtual_gid_maps = static:500
virtual_mailbox_base = /var/vmail/vmail01
virtual_mailbox_domains = 
proxy:mysql:/etc/postfix/mysql_virtual_mailbox_domains.cf
virtual_mailbox_maps = 
proxy:mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf

virtual_minimum_uid = 500
virtual_transport = dovecot
virtual_uid_maps = static:500

Regards,
Yaoxing


2010/12/23 23:13, Stan Hoeppner Wrote:

Wietse Venema put forth on 12/23/2010 6:10 AM:

4. Less than 20 postfix process (while limitation is explicitly set to 100)

Then, you are sending all mail through the same relay host. Why
are you sending mass mail through a relay host?

Wietse

It would appear my recommendation may merely mask (for a short time) a
symptom of the root problem, but not solve he problem itself.  Wietse's
insight seems to show that your IOPS issue is a result of the mail not
going out fast enough, and thus the disk is getting hammered by the
queue piling up.

Wieste is right.  You don't need faster storage, but more mails
delivered per second so they aren't piling up in the queue.  You need a
faster outbound path, or multiple of the same.

Apologies if my lesser insight/experience clogged this thread.

Re: postfix queue tuning

[Stan Hoeppner]

Wietse Venema put forth on 12/23/2010 6:10 AM:
>> 4. Less than 20 postfix process (while limitation is explicitly set to 100)
> 
> Then, you are sending all mail through the same relay host. Why
> are you sending mass mail through a relay host?
> 
>   Wietse

It would appear my recommendation may merely mask (for a short time) a
symptom of the root problem, but not solve he problem itself.  Wietse's
insight seems to show that your IOPS issue is a result of the mail not
going out fast enough, and thus the disk is getting hammered by the
queue piling up.

Wieste is right.  You don't need faster storage, but more mails
delivered per second so they aren't piling up in the queue.  You need a
faster outbound path, or multiple of the same.

Apologies if my lesser insight/experience clogged this thread.

-- 
Stan

Re: postfix queue tuning

[Stan Hoeppner]

Yaoxing put forth on 12/23/2010 4:45 AM:

> Is this what you're talking about?

Yes.

> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sda 148.63 27.55 6550.60 523033469 124353201092
> sda1 0.00 0.00 0.00 2524 116
> sda2 148.63 27.55 6550.59 523027626 124353101816
> sda3 0.00 0.00 0.01 2895 99160

This shows 148 transactions per second to device sda.  The average max
IOPS for any vendor's 7.2k rpm drive is ~150, a 15k drive hits ~300.
Your spool appears to be on partition sda2.  You are hitting the maximum
IOPS capability of that drive if it is a 7.2k drive.  If it is a 7.2k
drive, it would seem you need more spindles or an SSD.

I arrived at the 600 subscriber limit by multiplying 150 IOPS by 4
seconds, which is the interval you stated for successive mailings.  The
result is 600 subscribers maximum before exceeding the IOPS capability
of that drive.

-- 
Stan

Re: postfix queue tuning

[Wietse Venema]

Yaoxing Zhang:
> No special reasons, just trying to find a simple way to program. If this is
> the reason why it's so slow, how about using smtp instead? does it resolve
> the problem?

Please go back to the default settings, and report if the
system is still behaving slower than expected.

It is silly to make configuration changes, and then complain that
Postfix is not working well without sharing your changes (the
"postconf -n" output that you have been asked for).

Wietse

Re: postfix queue tuning

[Yaoxing Zhang]

No special reasons, just trying to find a simple way to program. If this is
the reason why it's so slow, how about using smtp instead? does it resolve
the problem?
On Dec 23, 2010 8:11 PM, "Wietse Venema"  wrote:

Re: postfix queue tuning

[Wietse Venema]

> 4. Less than 20 postfix process (while limitation is explicitly set to 100)

Then, you are sending all mail through the same relay host. Why
are you sending mass mail through a relay host?

Wietse

Re: postfix queue tuning

[Jeroen Geilman]

Post the output from postconf -n, and a relevant section of the mail logs.


--
J.

Re: postfix queue tuning

[Yaoxing]

It takes too long to show the complete result, but here's the first 
screen of

qshape active
 T  5 10 20 40 80 160 320 640 
1280 1280+
TOTAL 1000  0  0  0  0  0   0   0   
00  1000
gmail.com  254  0  0  0  0  0   0   0   
00   254
  hotmail.com  168  0  0  0  0  0   0   0   
00   168
yahoo.com   70  0  0  0  0  0   0   0   
0070
  comcast.net   14  0  0  0  0  0   0   0   
0014
  msn.com   12  0  0  0  0  0   0   0   
0012
  aol.com   11  0  0  0  0  0   0   0   
0011
sbcglobal.net   10  0  0  0  0  0   0   0   
0010
 yahoo.com.br9  0  0  0  0  0   0   0   
00 9
hotmail.co.uk9  0  0  0  0  0   0   0   
00 9
   web.de8  0  0  0  0  0   0   0   
00 8
  mail.ru8  0  0  0  0  0   0   0   
00 8
  free.fr8  0  0  0  0  0   0   0   
00 8
 yahoo.es7  0  0  0  0  0   0   0   
00 7
   hotmail.fr7  0  0  0  0  0   0   0   
00 7
   gmx.de6  0  0  0  0  0   0   0   
00 6
 juno.com6  0  0  0  0  0   0   0   
00 6
  cox.net5  0  0  0  0  0   0   0   
00 5
 live.com5  0  0  0  0  0   0   0   
00 5
 sympatico.ca5  0  0  0  0  0   0   0   
00 5

and for qshape incoming:
 T  5 10 20 40 80 160 320 640 
1280 1280+
TOTAL 1000  9  7 33 47 79 181 393 
2510 0
gmail.com  221  1  2  6  9 18  45 100  
400 0
  hotmail.com  153  1  1  4  9 15  32  63  
280 0
e.dealextreme.com  152  1  0  4  4  2  17  35  
890 0
yahoo.com   88  0  0  3  7  3  14  37  
240 0
  msn.com   12  0  0  0  0  0   1   6   
50 0
 yahoo.com.br   12  0  0  1  2  1   1   6   
10 0
  aol.com9  0  0  0  0  1   1   5   
20 0
  mail.ru8  0  0  0  0  1   2   3   
20 0
  free.fr7  0  0  0  0  2   1   4   
00 0
  comcast.net5  0  0  0  0  0   2   2   
10 0
   web.de4  0  0  0  0  1   0   2   
10 0
 yahoo.es4  0  0  0  1  0   1   2   
00 0
   hotmail.fr4  0  0  0  0  0   0   2   
20 0
 terra.com.br4  0  0  0  0  1   2   1   
00 0
 sympatico.ca4  0  0  0  0  1   0   2   
10 0
 ntlworld.com4  0  0  0  0  1   0   3   
00 0
bellsouth.net4  0  0  0  0  0   2   1   
10 0
   gmx.de3  0  0  0  0  1   0   2   
00 0
  ono.com3  0  0  0  0  0   0   3   
00 0


Regards,
Yaoxing


2010/12/23 18:57, Ralf Hildebrandt Wrote:

* Yaoxing:

Hi all,
I'm looking for some help of postfix server configuration. hope this is
the right place to ask.
I have a mail server running iRedMail (which is based on postfix). It
sends mails to our subscribers every 4s. I think this doesn't seem to be
a very heavy load. however, there're likely 140,000 mails congesting
after several days' running. So I tried qshape to analyse the queue, and
found that almost all mails are congesting in incoming queue, while
active queue reaches it's limit of 20,000 mails.

Please show the qshape output

Re: postfix queue tuning

[Ralf Hildebrandt]

* Yaoxing :
> Hi all,
> I'm looking for some help of postfix server configuration. hope this is
> the right place to ask.
> I have a mail server running iRedMail (which is based on postfix). It
> sends mails to our subscribers every 4s. I think this doesn't seem to be
> a very heavy load. however, there're likely 140,000 mails congesting
> after several days' running. So I tried qshape to analyse the queue, and
> found that almost all mails are congesting in incoming queue, while
> active queue reaches it's limit of 20,000 mails.

Please show the qshape output

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: postfix queue tuning

[Yaoxing]

Hi Ram,
I do have some more spare memory, but I'm afraid it doesn't resolve my 
problem.
Let's say, my active queue is filled with 20,000 mails, but mails are 
not going out but remains in memory. In this case if I increase active 
queue size, I just put more mails in memory, they still don't go out. 
What do you think?
I checked the directories, it's the incoming directory which is very 
big. if I use

find incomfing/ | wc -l
it tells me there are likely 130,000 mails in this directory.

Regards,
Yaoxing


2010/12/23 18:03, Ramprasad Wrote:

On Thu, 2010-12-23 at 11:59 +0800, Yaoxing wrote:

If you have a lot of RAM you could increase the active queue size.

Where are these mails getting delivered to ? Is they are going to the
same destination then you may require to increase the
default_destination_concurrency_limit

If you are choking on I/O iostat can indicate that
or else go to your spool directory and do a
du -s * .. see if some directory is taking really too long to du



Thanks
Ram

Re: postfix queue tuning

[Yaoxing]

Hi Stan,
Thank you very much for your adequate explanation. I made some comments
below.

Regards,
Yaoxing


2010/12/23 16:06, Stan Hoeppner:
> Yaoxing put forth on 12/22/2010 9:59 PM:
>
>> 3. 3.2MB/s disk IO write, 0.01MB/s read.
> MB/s throughput isn't usually a factor, but IOPS definitely can be.
> What's in the iostat tps column for the device your mail queues reside on?
Is this what you're talking about?
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 148.63 27.55 6550.60 523033469 124353201092
sda1 0.00 0.00 0.00 2524 116
sda2 148.63 27.55 6550.59 523027626 124353101816
sda3 0.00 0.00 0.01 2895 99160
> If your mail queue resides on a single mechanical disk spindle you may
> be running into an IOPS limit.  A single 7.2k rpm SATA disk can only
> sink approximately 150 seeks/sec (IOPS) under optimal conditions.  You
> didn't mention the size of your subscriber base, but you mentioned that
> you queue mail to those subscriuber addresses every 4 seconds. If using
> a single SATA disk as described above, you will run out of IOPS with
> ~600 subscribers.  If all of your queues and your log files, and your
> entire *nix system, resides on such a single disk, you'll run out of
> IOPS well below 600 subscribers.
I don't fully understand how this 600 subscribers is calculated. maybe I
should describe my situation like this:
It's a weekly promotion mail. no matter how many subscribers are there,
I send 1 email every 4sec.
Despite all I said above, do you mean if I have 600 subscribers, and I
send to all of them 1 mail every 4 sec (which is 600 mails/sec), I'll
run out of IOPS?
> After investigating, if this is indeed the cause of your queue
> performance problem, the solution is to put the Postfix queues on a
> separate storage device, optimally for performance and cost reasons, and
> SSD.
>
> Using a dedicated device for the queue files, a single 15k SAS disk can
> sink approximately 300 seeks/sec (IOPS) which would allow for
> approximately 1200 subscribers with a 4 sec interval between mailings.
> A RAID 10 set of 4 such drives will allow double that amount, or 2400
> subscribers.
>
> The cost of a good quality high performance SSD for Postfix queues is
> much less than 15k rpm SAS drives and far less than RAID 10, and will
> give you tens of thousands of IOPS, allowing for many thousands of
> subscribers, limited by SSD capacity.  If your problem is limited IOPS,
> I suggest putting your queues on one of these:
>
> http://www.newegg.com/Product/Product.aspx?Item=N82E16820227610
>
> If your total queue size is exceeding 40GB before mail can be flushed to
> the destinations, OCZ offers both 60GB and 90GB models for less than
> $200 USD.  IOPS performance for all 3 sizes of OCZ devices is similar,
> around 4k Random Write (Aligned): 45,000-50,000 IOPS.
>
> An SSD such as this is by far the best price/performance solution to
> such a queue IOPS problem, if that is indeed your queue performance issue.
I read more documents of postfix today, and according to my
understanding the active queue is in memory. Thus if I'm running out of
IOPS, I shouldn't have got a full active queue (tell me if I'm wrong).
But actually I do. So it's more likely a limitation of network or maybe
quantity of process. but either seems to be all right. That's why I'm
confused.

Re: postfix queue tuning

[Ramprasad]

On Thu, 2010-12-23 at 11:59 +0800, Yaoxing wrote:
> Hi all,
> I'm looking for some help of postfix server configuration. hope this is
> the right place to ask.
> I have a mail server running iRedMail (which is based on postfix). It
> sends mails to our subscribers every 4s. I think this doesn't seem to be
> a very heavy load. however, there're likely 140,000 mails congesting
> after several days' running. So I tried qshape to analyse the queue, and
> found that almost all mails are congesting in incoming queue, while
> active queue reaches it's limit of 20,000 mails. Mails rarely went to
> deferred queue. I think this means something reaches its limitation. So
> I checked the following features, and this is the result:
> 1. CPU is not busy at all
> 2. Almost 3GB memory left
> 3. 3.2MB/s disk IO write, 0.01MB/s read.
> 4. Less than 20 postfix process (while limitation is explicitly set to 100)
> There seems to be nothing wrong (or did I miss anything?). Can anybody
> provide some more information on locating the problem?
> Any help is appreciated.
> 


If you have a lot of RAM you could increase the active queue size. 

Where are these mails getting delivered to ? Is they are going to the
same destination then you may require to increase the
default_destination_concurrency_limit

If you are choking on I/O iostat can indicate that 
or else go to your spool directory and do a 
du -s * .. see if some directory is taking really too long to du



Thanks
Ram

Re: postfix queue tuning

[Stan Hoeppner]

Yaoxing put forth on 12/22/2010 9:59 PM:

> 3. 3.2MB/s disk IO write, 0.01MB/s read.

MB/s throughput isn't usually a factor, but IOPS definitely can be.
What's in the iostat tps column for the device your mail queues reside on?

If your mail queue resides on a single mechanical disk spindle you may
be running into an IOPS limit.  A single 7.2k rpm SATA disk can only
sink approximately 150 seeks/sec (IOPS) under optimal conditions.  You
didn't mention the size of your subscriber base, but you mentioned that
you queue mail to those subscriuber addresses every 4 seconds.  If using
a single SATA disk as described above, you will run out of IOPS with
~600 subscribers.  If all of your queues and your log files, and your
entire *nix system, resides on such a single disk, you'll run out of
IOPS well below 600 subscribers.

After investigating, if this is indeed the cause of your queue
performance problem, the solution is to put the Postfix queues on a
separate storage device, optimally for performance and cost reasons, and
SSD.

Using a dedicated device for the queue files, a single 15k SAS disk can
sink approximately 300 seeks/sec (IOPS) which would allow for
approximately 1200 subscribers with a 4 sec interval between mailings.
A RAID 10 set of 4 such drives will allow double that amount, or 2400
subscribers.

The cost of a good quality high performance SSD for Postfix queues is
much less than 15k rpm SAS drives and far less than RAID 10, and will
give you tens of thousands of IOPS, allowing for many thousands of
subscribers, limited by SSD capacity.  If your problem is limited IOPS,
I suggest putting your queues on one of these:

http://www.newegg.com/Product/Product.aspx?Item=N82E16820227610

If your total queue size is exceeding 40GB before mail can be flushed to
the destinations, OCZ offers both 60GB and 90GB models for less than
$200 USD.  IOPS performance for all 3 sizes of OCZ devices is similar,
around 4k Random Write (Aligned): 45,000-50,000 IOPS.

An SSD such as this is by far the best price/performance solution to
such a queue IOPS problem, if that is indeed your queue performance issue.

-- 
Stan

Re: postfix queue tuning

[John Adams]

Am 23.12.2010 04:59, schrieb Yaoxing:
> Hi all,
> I'm looking for some help of postfix server configuration. hope this is
> the right place to ask.
> I have a mail server running iRedMail (which is based on postfix). It
> sends mails to our subscribers every 4s. I think this doesn't seem to be
> a very heavy load. however, there're likely 140,000 mails congesting
> after several days' running. So I tried qshape to analyse the queue, and
> found that almost all mails are congesting in incoming queue, while
> active queue reaches it's limit of 20,000 mails. Mails rarely went to
> deferred queue. I think this means something reaches its limitation. So
> I checked the following features, and this is the result:
> 1. CPU is not busy at all
> 2. Almost 3GB memory left
> 3. 3.2MB/s disk IO write, 0.01MB/s read.
> 4. Less than 20 postfix process (while limitation is explicitly set to 100)
> There seems to be nothing wrong (or did I miss anything?). Can anybody
> provide some more information on locating the problem?
> Any help is appreciated.
> 

Does your postfix check whether the recipients' exist on your side?

Re: Postfix queue on ramdisk: Insufficient system storage

[Ram]

On Thu, 2010-07-22 at 04:50 -0500, Stan Hoeppner wrote:
> Patrick Ben Koetter put forth on 7/22/2010 2:11 AM:
> > * Stan Hoeppner :
> >> Wietse Venema put forth on 7/21/2010 2:22 PM:
> >>> Ram:
>  One server of ours just accepts the mails from clients and then relays
>  the mails to other servers. 
>  Since there is almost no mail queued on the server , I think it is will
>  be good to mount /var/spool/postfix on a tmpfs partition. 
> >>>
> >>> You will lose all mail in the queue when the system crashes.
> >>> I agree with Victor that this is a really bad idea.
> >>
> >> +3
> >>
> >> If you truly have a _need_ for a super fast Postfix queue, I suggest using 
> >> a
> >> good quality wear leveling SSD.  You'll get random I/O performance many 
> >> times
> >> greater than a 15k rpm disk, but with data persistence, unlike when using a
> >> ramdisk queue.  There are many fast good quality SSDs available in various
> >> capacities for between $100-200 USD, in standard 2.5" and 3.5" hard disk
> >> mounting form factors.
> > 
> > You can get about 150 msg/sec a 100k on a single Postfix instance if you use
> > a set of 10k rpm discs in a RAID 0 and server hardware.
> 
> If my math is correct, I believe Ram's relay server has a queue load of less
> than 15 msg/sec on average, which is easily handled by a single SATA disk.
> 
> 50,000/hr = 50,000/3600 = 13.88 msg/sec
> 
> Ram, why are you considering ramdisk or SSD for your Postfix queues given that
> a regular disk would seem to handle your load rather easily?  Or, is this more
> of a philosophical issue of not wanting to write anything to disk that isn't
> permanent?
> 

You are right.
If Postfix alone was running on this server will be able to handle
(50k-70k msgs/hr) with the given I/O. But there are other custom
functions running on this machine. 
I was just considering ramdisk , because that was the laziest way I
could get rid of unnecessary "IOPS" to disk. 

Anyway I think I will go by what all you folk say. No ramdisk for
postfix. 


Thanks for the inputs. 
Ram
 

Re: Postfix queue on ramdisk: Insufficient system storage

[Stan Hoeppner]

Patrick Ben Koetter put forth on 7/22/2010 2:11 AM:
> * Stan Hoeppner :
>> Wietse Venema put forth on 7/21/2010 2:22 PM:
>>> Ram:
 One server of ours just accepts the mails from clients and then relays
 the mails to other servers. 
 Since there is almost no mail queued on the server , I think it is will
 be good to mount /var/spool/postfix on a tmpfs partition. 
>>>
>>> You will lose all mail in the queue when the system crashes.
>>> I agree with Victor that this is a really bad idea.
>>
>> +3
>>
>> If you truly have a _need_ for a super fast Postfix queue, I suggest using a
>> good quality wear leveling SSD.  You'll get random I/O performance many times
>> greater than a 15k rpm disk, but with data persistence, unlike when using a
>> ramdisk queue.  There are many fast good quality SSDs available in various
>> capacities for between $100-200 USD, in standard 2.5" and 3.5" hard disk
>> mounting form factors.
> 
> You can get about 150 msg/sec a 100k on a single Postfix instance if you use
> a set of 10k rpm discs in a RAID 0 and server hardware.

If my math is correct, I believe Ram's relay server has a queue load of less
than 15 msg/sec on average, which is easily handled by a single SATA disk.

50,000/hr = 50,000/3600 = 13.88 msg/sec

Ram, why are you considering ramdisk or SSD for your Postfix queues given that
a regular disk would seem to handle your load rather easily?  Or, is this more
of a philosophical issue of not wanting to write anything to disk that isn't
permanent?

-- 
Stan

Re: Postfix queue on ramdisk: Insufficient system storage

[Patrick Ben Koetter]

* Stan Hoeppner :
> Wietse Venema put forth on 7/21/2010 2:22 PM:
> > Ram:
> >> One server of ours just accepts the mails from clients and then relays
> >> the mails to other servers. 
> >> Since there is almost no mail queued on the server , I think it is will
> >> be good to mount /var/spool/postfix on a tmpfs partition. 
> > 
> > You will lose all mail in the queue when the system crashes.
> > I agree with Victor that this is a really bad idea.
> 
> +3
> 
> If you truly have a _need_ for a super fast Postfix queue, I suggest using a
> good quality wear leveling SSD.  You'll get random I/O performance many times
> greater than a 15k rpm disk, but with data persistence, unlike when using a
> ramdisk queue.  There are many fast good quality SSDs available in various
> capacities for between $100-200 USD, in standard 2.5" and 3.5" hard disk
> mounting form factors.

You can get about 150 msg/sec a 100k on a single Postfix instance if you use
a set of 10k rpm discs in a RAID 0 and server hardware.

p...@rick


-- 
All technical questions asked privately will be automatically answered on the
list and archived for public access unless privacy is explicitely required and
justified.

saslfinger (debugging SMTP AUTH):

Re: Postfix queue on ramdisk: Insufficient system storage

[Stan Hoeppner]

Wietse Venema put forth on 7/21/2010 2:22 PM:
> Ram:
>> One server of ours just accepts the mails from clients and then relays
>> the mails to other servers. 
>> Since there is almost no mail queued on the server , I think it is will
>> be good to mount /var/spool/postfix on a tmpfs partition. 
> 
> You will lose all mail in the queue when the system crashes.
> I agree with Victor that this is a really bad idea.

+3

If you truly have a _need_ for a super fast Postfix queue, I suggest using a
good quality wear leveling SSD.  You'll get random I/O performance many times
greater than a 15k rpm disk, but with data persistence, unlike when using a
ramdisk queue.  There are many fast good quality SSDs available in various
capacities for between $100-200 USD, in standard 2.5" and 3.5" hard disk
mounting form factors.

-- 
Stan

Re: Postfix queue on ramdisk: Insufficient system storage

[Wietse Venema]

Ram:
> One server of ours just accepts the mails from clients and then relays
> the mails to other servers. 
> Since there is almost no mail queued on the server , I think it is will
> be good to mount /var/spool/postfix on a tmpfs partition. 

You will lose all mail in the queue when the system crashes.
I agree with Victor that this is a really bad idea.

Wietse

Re: Postfix queue on ramdisk: Insufficient system storage

[Victor Duchovni]

On Wed, Jul 21, 2010 at 06:39:07AM -0400, Wietse Venema wrote:

> > One server of ours just accepts the mails from clients and then relays
> > the mails to other servers. 
> > Since there is almost no mail queued on the server , I think it is will
> > be good to mount /var/spool/postfix on a tmpfs partition. 

[ Following up on the OP's post second-hand, not Wietse's ]

Also don't ask for trouble (memory exhaustion and/or email loss) by
running Postfix on ramdisk. It is not a good idea. Your intuition about
the value of this is misguided.

-- 
Viktor.