mikemccand commented on a change in pull request #2052:
URL: https://github.com/apache/lucene-solr/pull/2052#discussion_r518846067
##########
File path: lucene/misc/src/java/org/apache/lucene/store/DirectIODirectory.java
##########
@@ -164,15 +149,16 @@ public IndexOutput createOutput(String name, IOContext
context) throws IOExcepti
private long fileLength;
private boolean isOpen;
- public NativeUnixIndexOutput(Path path, String name, int bufferSize)
throws IOException {
- super("NativeUnixIndexOutput(path=\"" + path.toString() + "\")", name);
- //this.path = path;
- final FileDescriptor fd = NativePosixUtil.open_direct(path.toString(),
false);
- fos = new FileOutputStream(fd);
- //fos = new FileOutputStream(path);
- channel = fos.getChannel();
- buffer = ByteBuffer.allocateDirect(bufferSize);
- this.bufferSize = bufferSize;
+ @SuppressForbidden(reason = "com.sun.nio.file.ExtendedOpenOption: Direct I/O
with FileChannel requires the use of internal proprietary API
ExtendedOpenOption.DIRECT")
+ public DirectIOIndexOutput(Path path, String name) throws IOException {
+ super("DirectIOIndexOutput(path=\"" + path.toString() + "\")", name);
+
+ int blockSize = Math.toIntExact(Files.getFileStore(path).getBlockSize());
+ bufferSize = Math.addExact(blockSize, blockSize - 1);
+ buffer = ByteBuffer.allocateDirect(bufferSize).alignedSlice(blockSize);
Review comment:
Hmm I wonder why we got away without using `.alignedSlice` before?
##########
File path: lucene/CHANGES.txt
##########
@@ -156,6 +156,9 @@ Improvements
* LUCENE-9531: Consolidated CharStream and FastCharStream classes: these have
been moved
from each query parser package to org.apache.lucene.queryparser.charstream
(Dawid Weiss).
+* LUCENE-8982: Make NativeUnixDirectory pure java with FileChannel direct IO
flag,
Review comment:
Will this be Lucene 9.0 only?
The `com.sun.nio.file.ExtendedOpenOption.DIRECT` was added as of JDK 10
right?
Since Lucene 8.x still allows Java 8.x (and Lucene 9.x will require Java
11.x minimum), I think this improvement must be Lucene 9.x only?
##########
File path: lucene/misc/src/java/org/apache/lucene/store/DirectIODirectory.java
##########
@@ -66,45 +66,32 @@
*
* @lucene.experimental
*/
-public class NativeUnixDirectory extends FSDirectory {
+public class DirectIODirectory extends FSDirectory {
// TODO: this is OS dependent, but likely 512 is the LCD
private final static long ALIGN = 512;
private final static long ALIGN_NOT_MASK = ~(ALIGN-1);
-
- /** Default buffer size before writing to disk (256 KB);
- * larger means less IO load but more RAM and direct
- * buffer storage space consumed during merging. */
-
- public final static int DEFAULT_MERGE_BUFFER_SIZE = 262144;
/** Default min expected merge size before direct IO is
* used (10 MB): */
public final static long DEFAULT_MIN_BYTES_DIRECT = 10*1024*1024;
- private final int mergeBufferSize;
private final long minBytesDirect;
private final Directory delegate;
/** Create a new NIOFSDirectory for the named location.
*
* @param path the path of the directory
- * @param lockFactory to use
- * @param mergeBufferSize Size of buffer to use for
- * merging. See {@link #DEFAULT_MERGE_BUFFER_SIZE}.
* @param minBytesDirect Merges, or files to be opened for
* reading, smaller than this will
* not use direct IO. See {@link
* #DEFAULT_MIN_BYTES_DIRECT}
+ * @param lockFactory to use
* @param delegate fallback Directory for non-merges
* @throws IOException If there is a low-level I/O error
*/
- public NativeUnixDirectory(Path path, int mergeBufferSize, long
minBytesDirect, LockFactory lockFactory, Directory delegate) throws IOException
{
+ public DirectIODirectory(Path path, long minBytesDirect, LockFactory
lockFactory, Directory delegate) throws IOException {
super(path, lockFactory);
- if ((mergeBufferSize & ALIGN) != 0) {
- throw new IllegalArgumentException("mergeBufferSize must be 0 mod " +
ALIGN + " (got: " + mergeBufferSize + ")");
- }
- this.mergeBufferSize = mergeBufferSize;
Review comment:
Hmm does this mean we are losing the byte buffering during merging?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
