pgandhi999 commented on issue #22806: [SPARK-25250][CORE] : On successful completion of a task attempt on a parti… URL: https://github.com/apache/spark/pull/22806#issuecomment-450914469 @Ngone51 I see your point, and yes the key difference is that the PR #21131 marks partition as completed across all task attempts for the same stage attempt whereas this PR does that for all task attempts across all the stage attempts for the corresponding stage. We can extend the same behaviour here, but as far as killing running task attempts is concerned, that saves a lot of resources for long running tasks which might be consuming resources even though they have become redundant. I could be wrong here as well so let me know your thoughts. Thank you.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
