spapin opened a new pull request, #18738: URL: https://github.com/apache/pinot/pull/18738
## Why `MutableJsonIndexImpl#add` extends `_docIdMapping` for a document's flattened records, then builds the posting lists and increments `_nextFlattenedDocId`. While building those posting lists it maintains a running `_bytesSize` (a memory heuristic used to cap the mutable index) via Guava's `Utf8.encodedLength(...)`. `Utf8.encodedLength` throws `IllegalArgumentException` when a string contains an unpaired UTF-16 surrogate. Because it's called inside `computeIfAbsent` — after `_docIdMapping` has already been extended but before `_nextFlattenedDocId` is incremented — a single malformed value aborts the build loop and leaves `_docIdMapping` and `_nextFlattenedDocId` permanently out of sync. `_nextDocId` still advances in the `finally` block, so every subsequent document's `json_match` result is silently shifted by one flattened doc id. This behavior lasts until the segment is flushed. The non-mutable json indices do not estimate size, so its ids are in sync. ## Fix Route the size accounting through an `estimateLength` helper that falls back to `String.getBytes(UTF_8).length` (which never throws — malformed input is replaced) when `Utf8.encodedLength` throws. `_bytesSize` is only a heuristic, so an approximate length for a malformed string is acceptable; aborting indexing is not. ## Testing Added `testAddRecordWithUnpairedSurrogateDoesNotShiftDocIds`: indexes three records where the middle value is an unpaired surrogate (`\uD800`) and asserts `name='first'` → doc 0 and `name='third'` → doc 2. Fails before this change (doc ids shifted), passes after. Fixes #18737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
