Vamsi-klu opened a new pull request, #19088: URL: https://github.com/apache/pinot/pull/19088
## Why `MutableSegmentImpl.index()` is not transactional. On exception mid-row (dictionary/forward/agg), some columns were updated and others were not. `RealtimeSegmentDataManager` catches the error and **continues consuming** on the same mutable segment, so corruption becomes durable until seal fails (`IndexOutOfBoundsException`) or queries lie. Production effect: a single bad row can **poison a consuming partition** (seal failure, stuck consumption, data loss risk). ## Impact - **Mutable segments stay sealable** after transform/index errors. - **Per-row consistency** across dictionaries/forward indexes (complete-the-row with defaults/nulls instead of half-written docs). - **Observable quality**: incomplete/errored rows are metered rather than silent. - Also clears the stale multi-value + `aggregateMetrics` FIXME (aggregation is already disabled/rejected for MV dims). ## How - Fail-soft path in `index()` / `updateDictionary` / `addNewRow`: on failure, fill remaining columns with field default/null values for that docId and advance doc count consistently. - Treat forward-index failure as row-level incomplete (do not silently skip only the forward index). - Meter incomplete rows; avoid full dictionary transactional rollback (shared dict + concurrent readers). ## Test plan - [x] Extended `IndexingFailureTest` for mid-row failure → segment remains consistent / sealable. - [ ] `./mvnw -pl pinot-segment-local -am -Dtest=IndexingFailureTest -Dsurefire.failIfNoSpecifiedTests=false test` - [ ] Spot-check with ingestion aggregation bad input after #16317 lands (linked). ## Related fixes: #16316 related: #16317 ## Reviewers Suggested: Jackie-Jiang (issue guidance on fail-soft defaults) --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes — Grok Build (xAI) Generated-by: Grok Build (xAI) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
