[
https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-2795:
---------------------------------------
Attachment: LUCENE-2795.patch
Hi Varun!
Thanks for checking back here...
I ran this directory on Linux and hit a number of test failures, from
two bugs in the directory impl. First,
NativeUnixIndexInput.fileLength() failed to include bytes still in the
buffer; second, on clone, if the current II was positioned at EOF
and EOF was on a (512 byte) page boundary then we'd hit
a false IOE. I fixed those...
I also changed NativeUnixDir to take a delegate Dir which we fall back
to when we don't want to use direct IO; this enabled removing of
open_normal since we just use the "normal" java Dir impls for this.
To run all tests on Linux I do this:
{noformat}
export
CLASSPATH=/lucene/unixdir/lucene/build/contrib/misc/lucene-misc-4.0-SNAPSHOT.jar
export LD_LIBRARY_PATH=/lucene/unixdir/lucene/build/native:/usr/local/lib
ant test-core -Dtests.directory=NativeUnixDirectory
{noformat}
(Change the path to your full path... or maybe use relative path but
that could be dangerous since tests may change the CWD... not sure).
It also requires a temporary ctor in the directory taking only File
and using FSDirectory.open for the delegate directory.
Finally I added another setting, minBytesDirect: if the file to open
(in openInput), or expected merge size (in createOutput), is smaller
than this, then we don't use direct IO.
To test this... I temporarily fixed the dir to always use direct IO,
and tests (eventually: ~75 minutes!) passed, except for some Solr tests which
look
like timeout problems. I temporarily dropped the default
buffer size to 4 KB (from 256 KB) else the JVM ran out of direct
buffer space.
I'll test on OS X as well...
> Genericize DirectIOLinuxDir -> UnixDir
> --------------------------------------
>
> Key: LUCENE-2795
> URL: https://issues.apache.org/jira/browse/LUCENE-2795
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/store
> Reporter: Michael McCandless
> Assignee: Varun Thacker
> Labels: gsoc2011, lucene-gsoc-11, mentor
> Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
> LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
> LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch
>
>
> Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to
> use it for indexWriter and not IndexReader (searching). It's a trap.
> But, once we do LUCENE-2793, we can make it fully general purpose because
> then a single native Dir impl can be used.
> I'd also like to make it generic to other Unices, if we can, so that it
> becomes UnixDirectory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
