Copilot commented on code in PR #18738:
URL: https://github.com/apache/pinot/pull/18738#discussion_r3469857919
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/json/MutableJsonIndexImpl.java:
##########
@@ -134,19 +135,29 @@ private void addFlattenedRecords(List<Map<String,
String>> records) {
// Put both key and key-value into the posting list. Key is useful for
checking if a key exists in the json.
String key = entry.getKey();
_postingListMap.computeIfAbsent(key, k -> {
- _bytesSize += Utf8.encodedLength(key);
+ _bytesSize += estimateLength(key);
return new RoaringBitmap();
}).add(_nextFlattenedDocId);
String keyValue = key + JsonIndexCreator.KEY_VALUE_SEPARATOR +
entry.getValue();
_postingListMap.computeIfAbsent(keyValue, k -> {
- _bytesSize += Utf8.encodedLength(keyValue);
+ _bytesSize += estimateLength(keyValue);
return new RoaringBitmap();
}).add(_nextFlattenedDocId);
}
_nextFlattenedDocId++;
}
}
+ private static int estimateLength(String value) {
+ // Utf8.encodedLength counts bytes without allocating but throws on
malformed strings (e.g. unpaired surrogates).
+ // _bytesSize is only a heuristic, so fall back to getBytes (which never
throws) instead of failing.
+ try {
+ return Utf8.encodedLength(value);
+ } catch (IllegalArgumentException e) {
+ return value.getBytes(StandardCharsets.UTF_8).length;
+ }
+ }
Review Comment:
`estimateLength()` falls back to `value.getBytes(UTF_8)` on malformed
UTF-16. That allocates a full byte[] just to compute a heuristic size, which
can add significant transient memory/GC pressure for large values. Since
`_bytesSize` is only a heuristic, consider using a conservative non-allocating
estimate in the fallback path instead of materializing the UTF-8 bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
