vsop-479 commented on code in PR #16245:
URL: https://github.com/apache/lucene/pull/16245#discussion_r3556950880
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene103/blocktree/IntersectTermsEnumFrame.java:
##########
@@ -190,18 +197,32 @@ void load(TrieReader.Node node) throws IOException {
suffixesReader.reset(suffixBytes, 0, numSuffixBytes);
int numSuffixLengthBytes = ite.in.readVInt();
- final boolean allEqual = (numSuffixLengthBytes & 0x01) != 0;
+ allEqual = (numSuffixLengthBytes & 0x01) != 0;
numSuffixLengthBytes >>>= 1;
- if (suffixLengthBytes.length < numSuffixLengthBytes) {
- suffixLengthBytes = new byte[ArrayUtil.oversize(numSuffixLengthBytes,
1)];
- }
- if (allEqual) {
- Arrays.fill(suffixLengthBytes, 0, numSuffixLengthBytes,
ite.in.readByte());
+ if (isLeafBlock) {
+ if (allEqual) {
+ suffixLength = numSuffixLengthBytes;
+ } else {
+ final int numLongs = numSuffixLengthBytes;
+ suffixLengths = new int[entCount];
+ long[] longs = new long[numLongs];
+ for (int i = 0; i < numLongs; i++) {
+ longs[i] = ite.in.readLong();
+ }
+ // TODO: read and decode one long only when it is needed.
+ Simple64.decodeAll(longs, 0, suffixLengths, 0, entCount);
Review Comment:
> So this means we pre-decode all term suffix lengths for the terms in the
block, up front?
Yes, the Simple64 path eagerly decodes the suffix lengths into an `int[]`.
> Vs before when we would incrementally .readVInt() on iterating each term
in the block?
The previous VInt path also loaded the whole encoded suffix-length bytes
into `suffixLengthBytes`, then `suffixLengthsReader.readVInt()` incrementally
while scanning.
So this does not change the IO/materialization boundary for the encoded
data; the main difference is that Simple64 also decodes the lengths up front
instead of decoding each VInt as the block is scanned.
We could later decode one long only when we need to scan its corresponding
suffixLengths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
