[
https://issues.apache.org/jira/browse/LUCENE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-1121:
---------------------------------------
Attachment: LUCENE-1121.patch
Attached patch. All tests pass ... but, I don't think we should
commit this.
I ran performance tests across several platforms. All times are best
of 3 runs, indexing first 200K docs of Wikipedia. I used
SerialMergeScheduler for these tests so I could more easily measure
the impact on merging as well:
Linux (2.6.22), ReiserFS on RAID5 array: 528.1 sec vs 537.0 sec ( 1.7% faster)
Mac OS X 10.4 on RAID0 array: 402.6 sec vs 405.0 sec ( 0.6% faster)
Windows Server 2003 R64 on RAID0 array: 472.3 sec vs 752.6 sec (59.3% SLOWER)
I was rather stunned by the result on Windows Server 2003; I ran that
test twice to be sure. It's really true. My only guess is write
caching (which is turned on for this drive) is somehow not used when
using transferTo.
So then I made a standalone test that creates a big file (you specify
the size X 10 MB), and then copies that big file using transferTo and
then using an intermediate 64 KB buffer. Results below:
OS X 10.4 on external firewire drive: FASTER
create 500 MB file... 31689 msec
transferTo... 31947 msec
create 500 MB file... 31412 msec
buffer... 31215 msec
SLOWER 2.345%
OS X 10.4 on 4-drive RAID 0 array
create 500 MB file... 2409 msec
transferTo... 2449 msec
create 500 MB file... 2366 msec
buffer... 2649 msec
FASTER 7.55%
Linux 2.6.22 on single SATA drive, ext3
create 500 MB file... 12841 msec
transferTo... 12438 msec
create 500 MB file... 11219 msec
buffer... 12003 msec
SLOWER 3.624%
Linux 2.6.22 on 6-drive RAID 5 array, ext3
create 500 MB file... 9647 msec
transferTo... 9107 msec
create 500 MB file... 9092 msec
buffer... 10089 msec
FASTER 9.733%
Windows Server 2003 64 R2, single NTFS internal SATA drive
create 500 MB file... 32485 msec
transferTo... 38922 msec
create 500 MB file... 33484 msec
buffer... 1375 msec
SLOWER 2,730.691%
Windows XP Pro SP2, laptop hard drive
create 200 MB file... 20159 msec
transferTo... 17515 msec
create 200 MB file... 24265 msec
buffer... 18397 msec
FASTER 4.794%
Bottom line is: FileChannel.transferTo is not always a win, and, can
be a catastrophic loss. I think we should stick with tried & true,
simple, buffer copying, at least for now...
> Use nio.transferTo when copying large blocks of bytes
> -----------------------------------------------------
>
> Key: LUCENE-1121
> URL: https://issues.apache.org/jira/browse/LUCENE-1121
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: LUCENE-1121.patch, LUCENE-1121.patch
>
>
> When building a CFS file, and also when merging stored fields (and
> term vectors, with LUCENE-1120), we copy large blocks of bytes at
> once.
> We currently do this with an intermediate buffer.
> But, nio.transferTo should be somewhat faster on OS's that offer low
> level IO APIs for moving blocks of bytes between files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
