Jackie-Jiang commented on issue #8641: URL: https://github.com/apache/pinot/issues/8641#issuecomment-1122898428
@kkrugler IMO we should definitely fail the task if some segments are not properly generated. If we only log some warning/error but let the task succeed, very likely user will ignore them and get unexpected behavior. To keep the behavior similar to the current behavior, we can add them in multiple PRs: 1. Track all the failed segments into a list, but keep the task running. After all the segments are processed (succeeded or failed), fail the task if there are failed segments but keep the segment files in the output dir 2. Add the flag `terminateOnFailure`, and cancel/interrupt all the remaining segments on failure 3. We may also add a flag `cleanupOnFailure` to configure whether we want to clean up the output dir when the task failed. We do need to be careful here because the output dir might contain files not generated by this task, and we should not delete them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
