OK. Here's my CPU and memory benchmark findings
with my headersonly "fix" applied to my python
test script (which contains 90% TMDA original
code)
First of all, let me describe to you the function
of my script. It basically does three things:
1.) Read a message on STDIN.
2.) Parse the message's headers using the python
email module, then try to match the header
from address to an automatically generated
(from a flat text file) CDB database.
3.) Either exit or print the message to STDOUT.
I'm not going to go into the details of the script's
purpose. Suffice it to say that for all of the below
tests, this script WAS parsing the email's headers,
and WAS outputing the email, in full, to STDOUT.
My benchmark environment is this:
--------------------
MACHINE SPECS
--------------------
P-III 1.2Ghz IBM A30p laptop. ATA-100 HD, rotational
speed unknown.
386M of PC133 unbuffered laptop SDRAM RAM.
FreeBSD 5.1-RELEASE
Running KDE at the time of benchmarking
--------------------
TEST MESSAGE
--------------------
For the purpose of all below tests, I created a
15.8 Meg test message called 'test-message.txt'.
--------------------
BENCHMARK TYPES
--------------------
I performed two types of benchmarks:
1.) Time for process to complete
2.) Process Memory usage
--------------------
MEMORY USAGE BM
--------------------
For this test, I used the following command:
cat test-message.txt | ./block-forged-sender.py -s
And in "block-forged-sender.py" I placed:
import time
sys.sleep(3000)
Then, once the program was running (or sleeping,
depending on how you look at it), I would swap
to another terminal window and run:
ps auxwww | grep block-forged
--------------------
MEMORY USAGE RESULTS
--------------------
First, the results of loading the entire 15.8M
message into memory:
root 24569 53.9 9.1 36640 35256 p7 S+ 9:58PM 0:01.51 python
./block-forged-sender.py -s
For those of you unfamiliar with FreeBSD 'ps' output,
look at these numbers: 36640 35256
The above shows the memory footprint in Kb. My 15.8M
test message has created a 36.6M memory footprint!
Now, the results of loading JUST the headers into
memory, and passing the body of the message though:
root 24580 3.5 1.1 5628 4216 p7 S+ 9:59PM 0:00.10 python
./block-forged-sender.py -s
This is a dramatic improvement. My script now only
uses 5.6M of memory.
--------------------
TIME USAGE BM
--------------------
For this test, I used the following command:
/usr/bin/time -h ./time-test
"time-test" is an /bin/sh shell script:
------ BEGIN SHELL SCRIPT ------
#!/bin/sh
cat 'test-message.txt' | ./block-forged-sender.py > output.txt
------ END SHELL SCRIPT ------
I used this shell script so that the entire pipline's
execution time would be measured by the 'time' command,
not just the execution time of the 'cat' command.
I compared output.txt with the original test-message.txt,
and in both tests they were 100% identical.
--------------------
TIME USAGE RESULTS
--------------------
First, the results while loading the entire email into
memory, then writing it to disk:
3.10s real 1.37s user 0.67s sys
The time-test script took 3.1 seconds to finish.
Next, the results while loading just the headers into
memory, and passing the rest of the message through from
STDIN to STDOUT:
2.62s real 1.04s user 0.56s sys
Loading just the headers into memory knocked about half a second
off the execution time. (16% reduction in execution time)
--------------------
BM CONCLUSION
--------------------
Loading only an email's headers into memory saves both memory
and CPU time. With a workstation like my laptop, the savings are
negligible. However, in a server environment, every second of
CPU and every Meg of RAM counts.
I think this change is a good idea, and I plan to submit a
patch ASAP. Maybe today, maybe early next week.
--
Jesse Guardiani, Systems Administrator
WingNET Internet Services,
P.O. Box 2605 // Cleveland, TN 37320-2605
423-559-LINK (v) 423-559-5145 (f)
http://www.wingnet.net
_________________________________________________
tmda-workers mailing list ([EMAIL PROTECTED])
http://tmda.net/lists/listinfo/tmda-workers
