Hi, This was discussed before somewhere (in bugzilla ?) although I could not find it now)
I noticed a rather slow processing of POP3 e-mail fetching on a PC with scsi disk. At each message download, I can hear a clicking sound coming from a SCSI disk suggesting a large amount of disk activity. Well, I can certainly understand that TB probably uses |fdatasync()| or |fsync()| to make sure the downloaded message reaches the disk. But I was curious since the sound I hear is a little too much for a single message download. I began wondering that maybe TB was using |fsync()| where |fdatasync()| was enough, and that may have slowed down download process unduly. Last time,I raised the issue, somehow I could not find the reference to |fsync| in my local comm-central source tree, and maybe I was not searching it correctly. Yesterday, I noticed that, using mxr.mozilla.org, that |fdatasync| is defined into |fsync| if it is not defined at compile time in sqlite3.c: >From mxr.mozilla.org: mozilla/db/sqlite3/src/sqlite3.c (View Hg log or Hg annotations) line 27053 -- #if !defined(fdatasync) mozilla/security/nss/lib/sqlite/sqlite3.c (View Hg log or Hg annotations) line 26107 -- #if !defined(fdatasync) mozilla/toolkit/components/osfile/modules/osfile_unix_back.jsm (View Hg log or Hg annotations) line 392 -- libc.declareLazyFFI(SysFile, "fdatasync", line 393 -- "fdatasync", ctypes.default_abi, Quoting the first entry: 27047 /* 27048 ** We do not trust systems to provide a working fdatasync(). Some do. 27049 ** Others do no. To be safe, we will stick with the (slightly slower) 27050 ** fsync(). If you know that your system does support fdatasync() correctly, 27051 ** then simply compile with -Dfdatasync=fdatasync 27052 */ 27053 #if !defined(fdatasync) 27054 # define fdatasync fsync 27055 #endif Fair enough. I use linux and this is 2014. fdatasync() ought to work. So I re-compiled C-C thunderbird after adding -Dfdatasync=fdatasync to CC and CXX macro and (re-)configure. And I did some benchmarking of download time: Test method: I sent 2K short e-mails (1024 x is 2+ KB, and 1024 x 5+ KB ) messages to a local test account, and measured the download time from the local dovecot pop3 server to this account by clicking the download message button of TB explicitly (TB was configured not to download messages automatically). I used a stop watch style program to measure the time and so I only give the second resolution data below. Download time of 2024 e-mails: Before the change (using fsync() in sqlite3) 39 sec. 22 sec. 33 sec 24 sec 32 sec (note the wide variation.) After the change (using -Dfdatasync=fdatasync) 25 sec 21 sec 23 sec 21 sec 22 sec (note the narrower variation.) It is kind of difficult to say that using |fdatasync| shortened the download time. This is because the time with |fsync| varies too much. Then I realized that the wide variation of execution time is exactly because TB was using |fsync()|. Even on my local machine, there are many daemons that write data to log files, and creating and sending 2K e-mails with the following command also cause I/O traffic between the tests. for i in $(seq 1 1024); do echo $i; /usr/bin/mail -s "test $i" mtest2 < /etc/rygel.conf ; /usr/bin/mail -s "test 2 - $i" mtest2 < /etc/rsyslog.conf ; done cf. /etc/rygel.conf is 5078 bytes long, and /etc/rsyslog.conf is 2632 bytes long on my PC So depending on the background I/O activity, blindly using |fsync()| instead of |fdatasync()| causes too much elapsed time variation. This is indeed felt as undue slowdown sometimes for real. More to the point. I repeat the above test in the following manner: Initial step. I leave TB window open. Repeat: Send 2K messages as above to a local account. - download 2K messages (and measure the time.) into Inbox folder - delete the downloaded 2K messages (basically they are moved from Inbox to Trash folder: I can see the movement by the progress counter at the lower-left corner of the window pane.) - compat Inbox folder - empty Trash folder If necessary, repeat above. Now with |fsync()| the deletion of 2K messages (moving them from Inbox to Trash) is very sluggish and again its timing is widely varied. (I should have measured THAT time). OTOH, with |fdatasync| the time to move 2K messages is steady and constant. Obviously, unnecessary use of |fsync| is causing sometimes unnecessary delay while the movement of messages continue. The deletion of 2K messages is much nicer experience with |fdatasync| since the counter increments steadily without any strange pause that was often noticed with |fsync| and I definitely think the speed is faster with |fdatasync|. With "downloading", I think the overhead of moving data via network interface (even with on local PC) is too much and masks the advantage of |fdatasync|. But with local DISK COPY of messages from one folder to the other (during deletion), the advantage of |fdatasync| is obvious. I tested TB thus compiled with |make mozmill| test suite and no new error or warnings are introduced. Given the observation, I would like to propose that -Dfdatasync=fdatasync be put into the CC and CXX command line arguments for linux compilation. Searching "fsync" find many hits and so |fsync| seems to be used where it is absolutely necessary for data integrity instead of |fdatasync|. Any thought? A TB user who submit patches from time to time. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform