KKcorps opened a new pull request, #20007: URL: https://github.com/apache/pinot/pull/20007
A segment build reads its input twice. The stats pass sizes the dictionaries and sets total docs, and the index pass writes the rows. Nothing compared the two passes. In production a purge rebuild indexed one row fewer than its stats pass counted, so the sorted index kept an invalid `(Integer.MAX_VALUE, Integer.MIN_VALUE)` range for a value with no rows, and every query on that value failed until retention dropped the segment. We still don't know why the passes disagreed, so this PR makes the build fail loudly instead. `SegmentIndexCreationDriverImpl.build()` fails when the rows indexed differ from the stats-pass total, and the error names both counts and the segment. `SingleValueSortedForwardIndexCreator.close()` fails on a dictionary id that got no rows instead of writing its fill values to disk. `SegmentDictionaryCreator` throws on a value missing from the dictionary instead of quietly mapping it to dictionary id 0. The checks sit in the general segment build, not just the purge task, because any record reader that returns different rows on its two passes can hit this. When a failed build also fails to close the half-built sorted index, that close error is attached as suppressed so the real cause stays on top. New tests cover a reader that drops one row on the second pass, an unused sorted dictionary id, and dictionary misses for every stored type. All of them fail without the fix. The full `pinot-segment-local` suite (6,271 tests) and the `pinot-core` query, sorted and segment tests (1,520) pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
