Rajappa Iyer wrote:
> http://www.sysadminmag.com/articles/2001/0107/0107a/0107a.htm
> Any obvious reasons why FreeBSD performed so poorly for these people?

Here is a repeat of my post to -advocacy:

-- Terry


The article is meaningless.

Too bad they titled it "Which OS is Fastest for High-
Performance Network Applications?" instead of "Which OS is
Fastest for MailEngine?".

The only implied caveat is the statement "Our customers
frequently ask us which operating system is best for
running our software" in paragraph 3 of the "Background"
section.  This should have been in bold type in the first


It makes a number of very large blunders, which are really
inexcusable, given that it tries to represent itself as a
fair and unbiased comparison.

These blunders are in the tuning of FreeBSD, the best
architecture for FreeBSD applications (one shich they did
not even try to consider), in their choice of which items
they could micro-benchmark would really be indicative of
real-world performance, and, finally, in their experimental

Here is a short list, off the top of my head:

1)      The mail server they were using doesn't come
        with any of these systems "out of the box".

2)      Threaded processes are vastly inferior to
        finite state automatons, when it comes to CPU
        utilization on single CPU systems, and even on
        multiple CPU systems, if there is async I/O
        that can be scheduled on multiple CPUs.

3)      FreeBSD turns of write caching on IDE drives, by
        default, in FreeBSD 4.3 and above; you can set
        it to be like Linux, Solaris, and Windows, if
        you don't care about your data.  On FreeBSD 4.2
        and below, Soft Updates are not enabled by
        default.  Either way, without tuning, you lose.

4)      IDE drives do not support tagged command queueing,
        except IBM DTLA drives, which are known to fail
        due to overheating and due to their electronics
        being too slow for their radial track density for
        interior tracks.

5)      Real servers with storage and I/O requirements
        use SCSI drives so they can benefit from tagged
        command queues, which allow I/O to be interleaved
        instead of serialized.

6)      No well designed mail server keeps all queue
        files in the same directory, unless it has been
        designed to run on a particular system where that
        is not a problem; this is a design portability
        issue, not a performance issue.

7)      Sendmail can handle 400,000 8k emails in a 24 hour
        period on a < 500MHz box, if it is properly set up
        and queue dispersal is optimally configured (e.g.
        with the patches from ftp://ftp.whistle.com/ ).

8)      "The most efficient asynchornous architecture" for
        an application is OS-dependent.

9)      There are more than 3 ways to skin a cat, or to
        architect a task.

10)     Sending an RSET instead of data measures only the
        connection setup and teardown speeds, and does not
        measure real throughput, and is not representative
        of real world behaviour, in which mail messages,
        when sent, contain data, and not just trivial
        addressing information.

11)     Mail servers which support the ESMTP PIPELINE ability
        have significantly higher throughput, even when just
        doing addressing.

12)     You can not "tweak" FreeBSD's network connection
        limits at run time; socket structurse, inpcb's,
        and tcpcb's are allocated via a zone allocator,
        prior to the system actually being started.  This
        zone can not be resized.  Without the patch I posted
        to make maxfiles boot-time tunable, FreeBSD can not
        increase the number of sockets and files that it can
        simultaneously handle, without a kernel recompile.
        Thus the "tweaking" used was useless.

13)     For each connection, there is a tcptmpl structure,
        which is used for keepalives.  This structure will
        consume one mbuf per connection; in addition, the
        average TCP window size will be 16k; so you will
        need 16k/2k (8) mbufs for custer pointers plus
        16k/256 (64) mbufs for the window data, plus one
        mbuf per connection, pplus one mbuf per connection,
        if you are setting options.  This means that you
        will potentially need 74 mbufs per connection you
        intend to support; without patching, this also is
        not tunable except at compile time, and the value
        was not tuned.

14)     The "average througput per network architecture" is
        extremely misleading, both because of the limited
        and inefficient architectures used in the test, and
        in using an average to "determine" which was "the
        best architecture for use on all OSs".  Per OS numbers
        would have been much more meaningful, since the final
        architecture was chosen based on the average, and not
        based on what was best for the OS being tested.

15)     Creation and deletion of large numbers of files in a
        single directory is rather meaningless; witness the
        moronic 'postmark' "benchmark", and the resulting
        discussion on several FreeBSD lists.  The "benchmark"
        was clearly designed to put forth a political agenda
        (in the context of the list discussion, this agenda
        was the pro-ReiserFS position, despite Reiser's use
        of several USL/Novell patents on Delayed Ordered
        Writes, without the permission of the patent holders,
        which render a pro-ResierFS argument as meaningless
        as arguing about the number of angels which can dance
        on the head of a pin).

16)     When you have an answer you want, all the evidence you
        gather will tend to support the conclusion you have
        already drawn, unless you are very, very careful to
        eliminate your assumed biases in your experimental
        design.  This is very hard to do, even when trained
        scientists are involved.

Feel free to pass this on to the magazine editors, the author,
or both; at the very least, it may teach them a little bit
about how to actually tune FreeBSD to get good numbers out of
it, instead of frobbing meaningless knobs and merely _believing_
they are tuning the system for higher performance.

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message

Reply via email to