jpountz commented on code in PR #12712:
URL: https://github.com/apache/lucene/pull/12712#discussion_r1370397139
##########
lucene/misc/src/java/org/apache/lucene/misc/index/BPIndexReorderer.java:
##########
@@ -991,4 +939,166 @@ static int readMonotonicInts(DataInput in, int[] ints)
throws IOException {
}
return len;
}
+
+ static class ForwardIndexSorter {
+
+ private static final int HISTOGRAM_SIZE = 256;
+ private static final int BUFFER_SIZE = 8192;
+ private static final int BUFFER_BYTES = BUFFER_SIZE * Long.BYTES;
+ private final Directory directory;
+ private final Bucket[] buckets = new Bucket[HISTOGRAM_SIZE];
+
+ private static class Bucket {
+ final ByteBuffersDataOutput fps = new ByteBuffersDataOutput();
+ final long[] buffer = new long[BUFFER_SIZE];
+ IndexOutput output;
+ int bufferUsed;
+ int blockNum;
+ long lastFp;
+ int finalBlockSize;
+
+ void addEntry(long l) throws IOException {
+ buffer[bufferUsed++] = l;
+ if (bufferUsed == BUFFER_SIZE) {
+ flush(output, false);
+ }
+ }
+
+ void flush(IndexOutput output, boolean isFinal) throws IOException {
Review Comment:
Remove the IndexOutput from flush as well?
##########
lucene/misc/src/java/org/apache/lucene/misc/index/BPIndexReorderer.java:
##########
@@ -991,4 +939,166 @@ static int readMonotonicInts(DataInput in, int[] ints)
throws IOException {
}
return len;
}
+
+ static class ForwardIndexSorter {
+
+ private static final int HISTOGRAM_SIZE = 256;
+ private static final int BUFFER_SIZE = 8192;
+ private static final int BUFFER_BYTES = BUFFER_SIZE * Long.BYTES;
Review Comment:
Maybe add some comments about the fact that we expect this sorter to need
~16MB of heap (I think?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
