Re: NioFile cache performance

2005-12-09 Thread Chris Hostetter
: I have seen this issue come up several times (perhaps the following is : an oversimplification): : Someone will suggest a performance enhancement and perhaps supply the : code. Then there will be a general discussion about the merits of the : change and the validity of the results, with question

Re: NioFile cache performance

2005-12-09 Thread DM Smith
ibles. And the minimum hardware requirement is a Win98 laptop, which many of our user's have.) -Original Message- From: DM Smith [mailto:[EMAIL PROTECTED] Sent: Friday, December 09, 2005 7:07 AM To: java-dev@lucene.apache.org Subject: Re: NioFile cache performance John Haxby wro

RE: NioFile cache performance

2005-12-09 Thread Robert Engels
y, December 09, 2005 4:24 AM To: java-dev@lucene.apache.org Subject: Re: NioFile cache performance Robert Engels wrote: > Using a 4mb file (so I could be "guarantee" the disk data would be in > the OS cache as well), the test shows the following results. Which OS? If it

RE: NioFile cache performance

2005-12-09 Thread Robert Engels
, 2005 7:07 AM To: java-dev@lucene.apache.org Subject: Re: NioFile cache performance John Haxby wrote: > Robert Engels wrote: > >> Using a 4mb file (so I could be "guarantee" the disk data would be in >> the OS cache as well), the test shows the following results. >

Re: NioFile cache performance

2005-12-09 Thread DM Smith
John Haxby wrote: Robert Engels wrote: Using a 4mb file (so I could be "guarantee" the disk data would be in the OS cache as well), the test shows the following results. Which OS? If it's Linux, what kernel version and distro? What hardware (disk type, controller etc). It's important

Re: NioFile cache performance

2005-12-09 Thread John Haxby
Robert Engels wrote: Using a 4mb file (so I could be "guarantee" the disk data would be in the OS cache as well), the test shows the following results. Which OS? If it's Linux, what kernel version and distro? What hardware (disk type, controller etc). It's important to know: I/O (and ca

RE: NioFile cache performance

2005-12-08 Thread Robert Engels
switches into the OS will be avoided. -Original Message-From: Paul Smith [mailto:[EMAIL PROTECTED]Sent: Thursday, December 08, 2005 1:54 PMTo: java-dev@lucene.apache.orgSubject: Re: NioFile cache performance Most of the CPU time is actually used during the

Re: NioFile cache performance

2005-12-08 Thread Paul Smith
  Most of the CPU time is actually used during the synchronization with multiple threads. I hacked together a version of MemoryLRUCache that used a ConcurrentHashMap from JDK 1.5, and it was another 50% faster ! At a minimum, if the ReadWriteLock class was modified to use the 1.5 facilities some si

RE: NioFile cache performance

2005-12-08 Thread Robert Engels
ugh for most applications. I will attempt to get some performance numbers using/not using NioFile performing actual Lucene queries. -Original Message- From: Robert Engels [mailto:[EMAIL PROTECTED] Sent: Thursday, December 08, 2005 10:37 AM To: Lucene-Dev Subject: NioFile cache perf

NioFile cache performance

2005-12-08 Thread Robert Engels
I finally got around to writing a testcase to verify the numbers I presented. The following testcase and results are for the lowest level disk operations. On my machine reading from the cache, vs. going to disk (even when the data is in the OS cache) is 30%-40% faster. Since Lucene makes ext