warrenzhu25 commented on PR #41083: URL: https://github.com/apache/spark/pull/41083#issuecomment-1552141121
> These looks like things which can be handled by appropriate configuration tuning ? The PR itself requires a bit more work if that is not a feasible direction (efficient cleanup, handling corner cases, etc). Case 2 can't be handled by existing config, there'll be other similar recoverable cases. Generally speaking, I think unregister all map output when fetch failed is too aggressive. So it's better to have config to control or disable such behavior. If the executor is really dead, the map output will be unregistered when removing executor, if executor is just experiencing temporary and recoverable hiccup, then unregister is too expensive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
