Hi,
This was discussed before somewhere (in bugzilla ?) although I could not
find it now)
I noticed a rather slow processing of POP3 e-mail fetching on a PC with
scsi disk.
At each message download, I can hear a clicking sound coming from a SCSI
disk suggesting a large amount of disk activity.
Well, I can certainly understand that TB probably uses |fdatasync()| or
|fsync()| to make sure the downloaded message reaches the disk.
But I was curious since the sound I hear is a little too much for a
single message download. I began wondering that maybe TB was using
|fsync()| where |fdatasync()| was enough, and that may have slowed down
download process unduly.
Last time,I raised the issue, somehow I could not find the reference to
|fsync| in my local comm-central source tree, and maybe I was not
searching it correctly.
Yesterday, I noticed that, using mxr.mozilla.org, that |fdatasync| is
defined into |fsync| if it is not defined at compile time
in sqlite3.c:
>From mxr.mozilla.org:
mozilla/db/sqlite3/src/sqlite3.c (View Hg log or Hg annotations)
line 27053 -- #if !defined(fdatasync)
mozilla/security/nss/lib/sqlite/sqlite3.c (View Hg log or Hg
annotations)
line 26107 -- #if !defined(fdatasync)
mozilla/toolkit/components/osfile/modules/osfile_unix_back.jsm (View
Hg log or Hg annotations)
line 392 -- libc.declareLazyFFI(SysFile, "fdatasync",
line 393 -- "fdatasync", ctypes.default_abi,
Quoting the first entry:
27047 /*
27048 ** We do not trust systems to provide a working fdatasync(). Some do.
27049 ** Others do no. To be safe, we will stick with the (slightly slower)
27050 ** fsync(). If you know that your system does support fdatasync()
correctly,
27051 ** then simply compile with -Dfdatasync=fdatasync
27052 */
27053 #if !defined(fdatasync)
27054 # define fdatasync fsync
27055 #endif
Fair enough. I use linux and this is 2014.
fdatasync() ought to work.
So I re-compiled C-C thunderbird after
adding -Dfdatasync=fdatasync to CC and CXX macro and
(re-)configure.
And I did some benchmarking of download time:
Test method:
I sent 2K short e-mails (1024 x is 2+ KB, and 1024 x 5+ KB ) messages to
a local test account, and measured the download time from the local
dovecot pop3 server to this account by clicking the download message
button of TB explicitly (TB was configured not to download messages
automatically).
I used a stop watch style program to measure the time and so I only give
the second resolution data below.
Download time of 2024 e-mails:
Before the change (using fsync() in sqlite3)
39 sec.
22 sec.
33 sec
24 sec
32 sec
(note the wide variation.)
After the change (using -Dfdatasync=fdatasync)
25 sec
21 sec
23 sec
21 sec
22 sec
(note the narrower variation.)
It is kind of difficult to say that using |fdatasync|
shortened the download time. This is because the time with |fsync|
varies too much. Then I realized that the wide variation of execution
time is exactly because TB was using |fsync()|.
Even on my local machine, there are many daemons that write data to log
files, and creating and sending 2K e-mails with the following command
also cause I/O traffic between the tests.
for i in $(seq 1 1024); do echo $i; /usr/bin/mail -s "test $i" mtest2 <
/etc/rygel.conf ; /usr/bin/mail -s "test 2 - $i" mtest2 <
/etc/rsyslog.conf ; done
cf.
/etc/rygel.conf is 5078 bytes long, and
/etc/rsyslog.conf is 2632 bytes long on my PC
So depending on the background I/O activity, blindly using |fsync()|
instead of |fdatasync()| causes too much elapsed time variation. This is
indeed felt as undue slowdown sometimes for real.
More to the point.
I repeat the above test in the following manner:
Initial step. I leave TB window open.
Repeat:
Send 2K messages as above to a local account.
- download 2K messages (and measure the time.) into Inbox folder
- delete the downloaded 2K messages (basically they are moved from
Inbox to Trash folder: I can see the movement by the progress counter at
the lower-left corner of the window pane.)
- compat Inbox folder
- empty Trash folder
If necessary, repeat above.
Now with |fsync()| the deletion of 2K messages (moving them from Inbox
to Trash) is very sluggish and again its timing is
widely varied. (I should have measured THAT time).
OTOH, with |fdatasync| the time to move 2K messages is steady and
constant. Obviously, unnecessary use of |fsync| is causing sometimes
unnecessary delay while the movement of messages continue.
The deletion of 2K messages is much nicer experience with |fdatasync|
since the counter increments steadily without any strange pause that was
often noticed with |fsync| and I definitely think the speed is faster
with |fdatasync|. With "downloading", I think the overhead of
moving data via network interface (even with on local PC) is too much
and masks the advantage of |fdatasync|. But with local DISK COPY of
messages from one folder to the other (during deletion), the advantage
of |fdatasync| is obvious.
I tested TB thus compiled with |make mozmill| test suite and no new
error or warnings are introduced.
Given the observation, I would like to propose
that -Dfdatasync=fdatasync be put into the CC and CXX command line
arguments for linux compilation.
Searching "fsync" find many hits and so |fsync| seems to be used where
it is absolutely necessary for data integrity instead of |fdatasync|.
Any thought?
A TB user who submit patches from time to time.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
