mcvsubbu commented on issue #4070: Handle errors in consuming from streams consistent across: especially across HL and LLRealtimeSegmentDataManagers URL: https://github.com/apache/incubator-pinot/issues/4070#issuecomment-524567041 Stopping consumption is not a good idea in most cases. Errors can be due to two reasons: 1. Pinot bug in transform/read operations: We should be catching these in unit and integration tests and staging clusters. So far, we have not encountered this case in production at Linkedin (or reported here as an issue) 2. Bad data in stream: We have encountered this multiple times at Linkedin, as well as reported here as an issue (In the last case, I think we had one where a single string column was very huge causing segment build to fail, starting consumption all over for the same segment, and thus consumer never made forward progress. We introduced a limit in response to this). In general, streams do not allow administrators to "remove" offending rows. Once a row is in the stream, it is in. The choice is on the consumer to skip the row and make forward progress, or get stuck on consuming the row. I believe the former choice is better, especially for analytics platforms like Pinot. Not counting one row in aggregate for a day or so (the offline component is likely to correct the data) is a much better alternative than stopping consumption, and having admins manually set offset so skip over the offending data. Note that in either case, we have loss of data . In LLC we may be able to do more manual manipulations to skip over exact offending row, but in HLC it is impossible. If there is more than one offending row spread across, then problem is even worse, and by the time the manual corrections are applied, the stream is heavily backed up (in a typical production scenario) and the downtime is high. Conclusion: It is best to move on after discarding the row.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
