[
https://issues.apache.org/jira/browse/LUCENE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609629#action_12609629
]
Jason Rutherglen commented on LUCENE-753:
-----------------------------------------
Core2Duo Windows XP JDK1.5.15. PooledPread for 4 threads and pool size 2 the
performance does not compare well to SeparateFile. PooledPread for 30 threads
does not improve appreciably over ClassicFile. If there were 30 threads, how
many RandomAccessFiles would there need to be to make a noticeable impact? The
problem I see with the pooled implementation is setting the global file
descriptor limit properly, will the user set this? There would almost need to
be a native check to see if what the user is trying to do is possible.
The results indicate there is significant contention in the pool code. The
previous tests used a pool size the same as the number of threads which is
probably not how most production systems look, at least the SOLR installations
I've worked on. In SOLR the web request thread is the thread that executes the
search, so the number of threads is determined by the J2EE server which can be
quite high. Unless the assumption is the system is set for an unusually high
number of file descriptors.
There should probably be a MMapDirectory test as well.
{noformat}
config: impl=PooledPread serial=false nThreads=4 iterations=100 bufsize=1024
poolsize=2 filelen=18110448
answer=53797223, ms=32715, MB/sec=221.4329573590096
config: impl=SeparateFile serial=false nThreads=4 iterations=100 bufsize=1024
poolsize=2 filelen=18110448
answer=53797223, ms=18687, MB/sec=387.6587574249478
config: impl=SeparateFile serial=false nThreads=30 iterations=100 bufsize=1024
poolsize=2 filelen=18110448
answer=403087371, ms=137871, MB/sec=394.0737646060448
config: impl=PooledPread serial=false nThreads=30 iterations=100 bufsize=1024
poolsize=2 filelen=18110448
answer=403087487, ms=526504, MB/sec=103.19265190767781
config: impl=ChannelPread serial=false nThreads=30 iterations=100 bufsize=1024
poolsize=2 filelen=18110448
answer=403087487, ms=624291, MB/sec=87.02887595688549
config: impl=ClassicFile serial=false nThreads=30 iterations=100 bufsize=1024
poolsize=2 filelen=18110448
answer=403087487, ms=587430, MB/sec=92.48990347786119
config: impl=PooledPread serial=false nThreads=30 iterations=100 bufsize=1024
poolsize=4 filelen=18110448
answer=403087487, ms=552971, MB/sec=98.25351419875544
{noformat}
> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>
> Key: LUCENE-753
> URL: https://issues.apache.org/jira/browse/LUCENE-753
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Reporter: Yonik Seeley
> Attachments: FileReadTest.java, FileReadTest.java, FileReadTest.java,
> FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java,
> FSIndexInput.patch, FSIndexInput.patch, lucene-753.patch
>
>
> As suggested by Doug, we could use NIO pread to avoid synchronization on the
> underlying file.
> This could mitigate any MT performance drop caused by reducing the number of
> files in the index format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
