Re: How disk I/O affect postfix performance ?

[Wietse Venema]

Yu (Irvin) Fan:
> Hi Wietse,
> 
> Thanks for the quick answer. Can I say that the postfix performance is
> affected by small file read/write speed of the disk?

Many email messages small. Therefore performance is dominated by
rotational and seek latencies (absent a large persistent buffer
between the kernel and the disk drive).

> Anyway, I have another related question. We have an after-queue content
> filter written in Perl. The email is fed to the filter using pipe as
> suggested in the following example in the documentation.
> 
>   smtp  inet  ...other stuff here, do not change...   smtpd
> -o content_filter
> =filter:dummy
> 
>   filterunix  -   n   n   -   10  pipe
> flags=Rq user=filter argv=/path/to/script -f ${sender} -- ${recipient}
> 
> 
> We notice that if the disk activities (by other processes in the box) went
> up there would be less # of filter scripts running. But at the same time the
> smtpd queuing doesn't seem to be affected that much.

The smtpd processes load the disk. Therefore, less I/O can be done
by the queue manager. Postfix has a limited safety mechanism that
works only when the mail load comes from a small number of clients.

To reduce disk load, receive less mail. That is, provide less
bandwidth to receive mail, or fewer smtpd processes.

> So we end up with
> backlog building in incoming queue. I want to understand does this happen
> naturally because spawning filter would require more system resource
> (loading, compiling, etc. even our filter is lightweight) so if the disk is
> slow it would be affected the most?

You would not have asked this question if you had looked at the CPU
load of your machine.

> or is there some kind of throttling
> control built in postfix (can't find anything about it in documentation
> though) by which the postfix qmgr would spawn less filters on purpose in
> order to make sure smtpd have enough resource to handle incoming emails
> cause we don't want to reject emails? If postfix does have throttling
> control what are the deciding factors it's looking at?

The queue manager works as fast as it can.

> btw, If I turn the filter into a daemon and feed the content through smtp
> would that help a lot?

It would use less CPU than creating a new process for every message,
but it matters only when the disk is fast enough. Which it is not
in the present case.

Wietse

Re: How disk I/O affect postfix performance ?

[Yu (Irvin) Fan]

Hi Wietse,

Thanks for the quick answer. Can I say that the postfix performance is
affected by small file read/write speed of the disk?

Anyway, I have another related question. We have an after-queue content
filter written in Perl. The email is fed to the filter using pipe as
suggested in the following example in the documentation.

  smtp  inet  ...other stuff here, do not change...   smtpd
-o content_filter
=filter:dummy

  filterunix  -   n   n   -   10  pipe
flags=Rq user=filter argv=/path/to/script -f ${sender} -- ${recipient}


We notice that if the disk activities (by other processes in the box) went
up there would be less # of filter scripts running. But at the same time the
smtpd queuing doesn't seem to be affected that much. So we end up with
backlog building in incoming queue. I want to understand does this happen
naturally because spawning filter would require more system resource
(loading, compiling, etc. even our filter is lightweight) so if the disk is
slow it would be affected the most? or is there some kind of throttling
control built in postfix (can't find anything about it in documentation
though) by which the postfix qmgr would spawn less filters on purpose in
order to make sure smtpd have enough resource to handle incoming emails
cause we don't want to reject emails? If postfix does have throttling
control what are the deciding factors it's looking at?

btw, If I turn the filter into a daemon and feed the content through smtp
would that help a lot?

Thanks

-Irvin

On Fri, Feb 13, 2009 at 4:45 AM, Wietse Venema  wrote:

> Wietse Venema:
> > Yu (Irvin) Fan:
> > > Hi,
> > >
> > > We're building a box to run two postfix instances to receive and send
> high
> > > volume of emails. According to the documentation it's better to run the
> two
> > > instances on separate disks for performance reason. I'm trying to
> understand
> > > how exactly does the disk I/O affect the postfix performance? By speed
> > > (bytes per second) or activities (# of read/write per second)?
> >
> > I have shocking information for you: it is none of the above.
> >
> > Postfix must write the message to stable storage, so that it
> > will not be lost after a system crash.
> >
> > For example, writing mail to a queue file requires multiple file
> > system updates:
> >
> > - allocate queue file inode (in inode bitmap etc.)
> > - allocate queue file blocks (in block bitmap etc.)
> > - update queue file blocks
> > - update queue file inode
> > - update directory file (for queue file name etc.)
> > - update directory file inode
> >
> > Each of these are in a different place in the file system.  Only
> > once all this information is updated, can Postfix claim that mail
> > is in stable storage.
> >
> > Thus the read/write speed is largely irrelevant for small email
> > messages. Performance is dominated by seek latency and rotational
> > latency.
>
> You can reduce the latencies by using a large non-volatile buffer
> (as is common with RAID systems). With the large non-volatile
> buffer, writes complete quickly. The hardware can sort the update
> order to minimize head movements.
>
>Wietse
>

Re: How disk I/O affect postfix performance ?

[Wietse Venema]

Wietse Venema:
> Yu (Irvin) Fan:
> > Hi,
> > 
> > We're building a box to run two postfix instances to receive and send high
> > volume of emails. According to the documentation it's better to run the two
> > instances on separate disks for performance reason. I'm trying to understand
> > how exactly does the disk I/O affect the postfix performance? By speed
> > (bytes per second) or activities (# of read/write per second)?
> 
> I have shocking information for you: it is none of the above.
> 
> Postfix must write the message to stable storage, so that it
> will not be lost after a system crash.
> 
> For example, writing mail to a queue file requires multiple file
> system updates:
> 
> - allocate queue file inode (in inode bitmap etc.)
> - allocate queue file blocks (in block bitmap etc.)
> - update queue file blocks
> - update queue file inode
> - update directory file (for queue file name etc.)
> - update directory file inode
> 
> Each of these are in a different place in the file system.  Only
> once all this information is updated, can Postfix claim that mail
> is in stable storage.
> 
> Thus the read/write speed is largely irrelevant for small email
> messages. Performance is dominated by seek latency and rotational
> latency.

You can reduce the latencies by using a large non-volatile buffer
(as is common with RAID systems). With the large non-volatile
buffer, writes complete quickly. The hardware can sort the update
order to minimize head movements.

Wietse

Re: How disk I/O affect postfix performance ?

[Wietse Venema]

Yu (Irvin) Fan:
> Hi,
> 
> We're building a box to run two postfix instances to receive and send high
> volume of emails. According to the documentation it's better to run the two
> instances on separate disks for performance reason. I'm trying to understand
> how exactly does the disk I/O affect the postfix performance? By speed
> (bytes per second) or activities (# of read/write per second)?

I have shocking information for you: it is none of the above.

Postfix must write the message to stable storage, so that it
will not be lost after a system crash.

For example, writing mail to a queue file requires multiple file
system updates:

- allocate queue file inode (in inode bitmap etc.)
- allocate queue file blocks (in block bitmap etc.)
- update queue file blocks
- update queue file inode
- update directory file (for queue file name etc.)
- update directory file inode

Each of these are in a different place in the file system.  Only
once all this information is updated, can Postfix claim that mail
is in stable storage.

Thus the read/write speed is largely irrelevant for small email
messages. Performance is dominated by seek latency and rotational
latency.

Wietse

> Let's say I have two hard disks. If I make a RAID0 array out of the two
> disks the overall speed is twice as the speed of a single drive (I know it's
> not exact twice the speed. just simplify it for discussion). If the postfix
> performance depends on the disk speed then running two instances on two
> separate disks or running two instances on one RAID0 array should not make
> big difference, right? But if it depends on disk activities then running on
> two separate disks is definitely better. I know no matter which case using
> two disks is the choice. But if I have other reason to use RAID0 I just want
> to know how much performance I lost in postfix?
> 
> Thanks
> 
> -Irvin

How disk I/O affect postfix performance ?

[Yu (Irvin) Fan]

Hi,

We're building a box to run two postfix instances to receive and send high
volume of emails. According to the documentation it's better to run the two
instances on separate disks for performance reason. I'm trying to understand
how exactly does the disk I/O affect the postfix performance? By speed
(bytes per second) or activities (# of read/write per second)?

Let's say I have two hard disks. If I make a RAID0 array out of the two
disks the overall speed is twice as the speed of a single drive (I know it's
not exact twice the speed. just simplify it for discussion). If the postfix
performance depends on the disk speed then running two instances on two
separate disks or running two instances on one RAID0 array should not make
big difference, right? But if it depends on disk activities then running on
two separate disks is definitely better. I know no matter which case using
two disks is the choice. But if I have other reason to use RAID0 I just want
to know how much performance I lost in postfix?

Thanks

-Irvin