[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546795#comment-13546795 ] Michael McCandless commented on LUCENE-2795: Woops, thanks Littlestar: the comment should say 256 KB. I'll fix. Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Core Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Fix For: 4.0-ALPHA Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546798#comment-13546798 ] Commit Tag Bot commented on LUCENE-2795: [branch_4x commit] Michael McCandless http://svn.apache.org/viewvc?view=revisionrevision=1430216 LUCENE-2795: fix wrong comment (KB not MB) Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Core Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Fix For: 4.0-ALPHA Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546799#comment-13546799 ] Commit Tag Bot commented on LUCENE-2795: [trunk commit] Michael McCandless http://svn.apache.org/viewvc?view=revisionrevision=1430214 LUCENE-2795: fix wrong comment (KB not MB) Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Core Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Fix For: 4.0-ALPHA Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546645#comment-13546645 ] Littlestar commented on LUCENE-2795: /** Default buffer size before writing to disk (256 MB); * larger means less IO load but more RAM and direct * buffer storage space consumed during merging. */ public final static int DEFAULT_MERGE_BUFFER_SIZE = 262144; in NativeUnixDirectory.java Does 256MB mistake ? 262144===256K Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Core Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Fix For: 4.0-ALPHA Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187045#comment-13187045 ] Michael McCandless commented on LUCENE-2795: OK tests pass (slowly: 46 minutes) on OS X as well. I think this is ready! Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117404#comment-13117404 ] Varun Thacker commented on LUCENE-2795: --- Hi, I think this patch is almost complete. It will be great if anyone can review it. Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[
https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070644#comment-13070644
]
Michael McCandless commented on LUCENE-2795:
Getting closer!
Instead of {{#define LINUX linux}} you can just do {{#define LINUX}}.
Can you rename forcedBufferSize - mergeBufferSize in all places?
(And also the default constant FORCED_...).
I don't think 0 should mean we fallback to BII's default; ie, just
remove that logic and always use mergeBufferSize, and fix the jdoc.
How come you're passing {{true}} as the readOnly arg to
open_direct/_normal in createOutput? Seems like this will fail? (Hmm
did you run tests w/ these changes?).
Genericize DirectIOLinuxDir - UnixDir
--
Key: LUCENE-2795
URL: https://issues.apache.org/jira/browse/LUCENE-2795
Project: Lucene - Java
Issue Type: Improvement
Components: core/store
Reporter: Michael McCandless
Assignee: Varun Thacker
Labels: gsoc2011, lucene-gsoc-11, mentor
Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch,
LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch
Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to
use it for indexWriter and not IndexReader (searching). It's a trap.
But, once we do LUCENE-2793, we can make it fully general purpose because
then a single native Dir impl can be used.
I'd also like to make it generic to other Unices, if we can, so that it
becomes UnixDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068636#comment-13068636 ] Michael McCandless commented on LUCENE-2795: I think NativeUnixDir should have a ctor that takes no buffer size, and uses the default? And merging should always use the forcedBufferSize (let's rename it to mergeBufferSize?), ie, never delegate to BII.bufferSize(context). Then, NativeUnixIndexInput/Output shouldn't have 2 ctors, only 1 (that takes bufferSize) and NativeUnixDir just passes the buffer size? Somehow, but this should be a different issue, this Dir impl should only provide the II/IO impls for use during merging; if it's not a merge it should delegate to another Dir impl. Maybe we pass a delegate to the ctor or something... Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067097#comment-13067097 ] Michael McCandless commented on LUCENE-2795: Can you touch up the javadocs? Ie it's more general now (not just linux), and uses direct io instead of Linux's O_DIRECT flag, and uses direct io for all merge input output. Make it clear that this is a general purpose dir impl... NativeUnixndexOutput should be NativeUnixIndexOutput. The createOutput function should also switch between normal vs direct, depending on MERGE or READ? I think we should have a default forced buffer size? The thing to keep in mind is how many open inputs/outputs we will have during merging, eg I think something like 5 or 8 files per merged segment plus another 5 or 8 for the resulting merged segment, so up to 88 file handles open. Maybe 256 KB default? We shouldn't by default use the BII's buffer size: it's way too small for direct IO. And then we should always use the forced buffer size in this dir... Tests pass for me on Fedora 13 linux! Next I'll try OS X. Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067105#comment-13067105 ] Michael McCandless commented on LUCENE-2795: Also, we should run some more tests here, on modern kernels, since the SEQUENTIAL fadvise flag is better implemented in kernels = 2.6.29; see this comment: http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html?showComment=1303235497682#c2572106601600642254 It could be, if kernel is new enough, we should not do DIRECT and instead do SEQUENTIAL. Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067146#comment-13067146 ] Michael McCandless commented on LUCENE-2795: OK, passes for me on OS X 10.6.6. I first had to install the Java developer package (jni.h was missing) from http://connect.apple.com Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[
https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066619#comment-13066619
]
Michael McCandless commented on LUCENE-2795:
OK, so now we create a FileDescriptor obj in C and return that, by
setting the fd field directly? This is because fadvise/posix_fadvise
on an already opened fd isn't portable enough? Ie we must open the
file w/ the flags, instead, for some OS's?
I don't think you need posix_fadvise.h at all? Can't you just
#include the right headers per OS, inside each OS's ifdef?
Missing some { } around single-statement if bodies. Can you move the
return NULL in if (class_ioex == NULL) return NULL; to the next
line, with { }?
Genericize DirectIOLinuxDir - UnixDir
--
Key: LUCENE-2795
URL: https://issues.apache.org/jira/browse/LUCENE-2795
Project: Lucene - Java
Issue Type: Improvement
Components: core/store
Reporter: Michael McCandless
Assignee: Varun Thacker
Labels: gsoc2011, lucene-gsoc-11, mentor
Attachments: LUCENE-2795.patch, LUCENE-2795.patch, LUCENE-2795.patch
Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to
use it for indexWriter and not IndexReader (searching). It's a trap.
But, once we do LUCENE-2793, we can make it fully general purpose because
then a single native Dir impl can be used.
I'd also like to make it generic to other Unices, if we can, so that it
becomes UnixDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[
https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13060331#comment-13060331
]
Varun Thacker commented on LUCENE-2795:
---
{quote}
Hey Varun - saw you asked for someone with a mac to run some code for you in
IRC but you popped off before I saw - what do you need? Just apply the patch
and run the tests?
{quote}
This patch will apply to the LUCENE2793 branch. Othewise in file :
lucene/contrib/misc/src/java/org/apache/lucene/store/NativePosixUtil.cpp after
line 117 inside the if add this line - {code} fcntl(fd, F_NOCACHE, 1); {code}
And then by running {code}ant build-native-unix{code} from the /contrib/misc
folder to check if it compiles successfully.
Thanks.
Genericize DirectIOLinuxDir - UnixDir
--
Key: LUCENE-2795
URL: https://issues.apache.org/jira/browse/LUCENE-2795
Project: Lucene - Java
Issue Type: Improvement
Components: core/store
Reporter: Michael McCandless
Assignee: Varun Thacker
Labels: gsoc2011, lucene-gsoc-11, mentor
Attachments: LUCENE-2795.patch
Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to
use it for indexWriter and not IndexReader (searching). It's a trap.
But, once we do LUCENE-2793, we can make it fully general purpose because
then a single native Dir impl can be used.
I'd also like to make it generic to other Unices, if we can, so that it
becomes UnixDirectory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13060216#comment-13060216 ] Mark Miller commented on LUCENE-2795: - Hey Varun - saw you asked for someone with a mac to run some code for you in IRC but you popped off before I saw - what do you need? Just apply the patch and run the tests? Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: core/store Reporter: Michael McCandless Assignee: Varun Thacker Labels: gsoc2011, lucene-gsoc-11, mentor Attachments: LUCENE-2795.patch Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023642#comment-13023642 ] Varun Thacker commented on LUCENE-2795: --- This is great for merging . Does IndexReader use the SEQUENTIAL flag for accessing the index ? Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: Store Reporter: Michael McCandless Assignee: Simon Willnauer Labels: gsoc2011, lucene-gsoc-11, mentor Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13024631#comment-13024631 ] Michael McCandless commented on LUCENE-2795: Today, we don't use any of these flags for searching/merging (that's what this issue and LUCENE-2793 will fix). Unfortunately, it looks like the best IO flags are going to vary depending on kernel version. The above patch was applied in kernel 2.6.29, and causes SEQUENTIAL to act like NOREUSE, and so we should use SEQUENTIAL for merging but not for searching, I think? We should experiment and see... Really, for searching we want the old SEQUENTIAL behavior (ie, do aggressive readahead, but use the normal eviction policy since that term may be searched on again). Hopefully NativeUnixDir will take care of all these os/kernel-version dependent flags for you. Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: Store Reporter: Michael McCandless Assignee: Simon Willnauer Labels: gsoc2011, lucene-gsoc-11, mentor Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023614#comment-13023614 ] Michael McCandless commented on LUCENE-2795: It looks like recent Linux kernels have better behavior with SEQUENTIAL flag: http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html?showComment=1303235497682#c2572106601600642254 If possible we should test on kernels after that patch was merged to see if passing SEQUENTIAL for merging prevents eviction of hot pages being used for searching... Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: Store Reporter: Michael McCandless Assignee: Simon Willnauer Labels: gsoc2011, lucene-gsoc-11, mentor Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2795) Genericize DirectIOLinuxDir - UnixDir
[ https://issues.apache.org/jira/browse/LUCENE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12979719#action_12979719 ] Michael McCandless commented on LUCENE-2795: https://issues.apache.org/jira/browse/LUCENE-2500?focusedCommentId=12979588page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12979588 has details on what flags to pass to OS X to bypass its buffer cache... Genericize DirectIOLinuxDir - UnixDir -- Key: LUCENE-2795 URL: https://issues.apache.org/jira/browse/LUCENE-2795 Project: Lucene - Java Issue Type: Improvement Components: Store Reporter: Michael McCandless Today DirectIOLinuxDir is tricky/dangerous to use, because you only want to use it for indexWriter and not IndexReader (searching). It's a trap. But, once we do LUCENE-2793, we can make it fully general purpose because then a single native Dir impl can be used. I'd also like to make it generic to other Unices, if we can, so that it becomes UnixDirectory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
