ajithme commented on issue #24142: [SPARK-27194][core] Job failures when task attempts do not clean up spark-staging parquet files URL: https://github.com/apache/spark/pull/24142#issuecomment-475000570 > You seem to be trying to create a new commit protocol that a single task can perform without regard for what other tasks are doing. That is just not going to work. No. I am trying to point the difference when dynamicPartitionOverwrite is false vs true. When False, file list is iterated to be moved. But when true entire folder is just renamed. Task commits are still independent, but job commit should consider task status. In case of retry tasks as output must not be overlapping we could have had separate file for retry and handled eliminating duplicate task outputs on job commit. I am just saying that as per current code, moving entire staging-dir partition output to final location regardless of its content (if any tasks were retired) is not good approach, it just works as retry tasks also have same file for output (hence avoids duplicate). Sorry if approach was wrong. But my concern here is, A abnormal task which has exited, should not block or fail the retry task otherwise the whole point of retry task is useless. Please suggest if you see a better approach to tackle the problem here
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
