zhongyu09 commented on pull request #31084: URL: https://github.com/apache/spark/pull/31084#issuecomment-756599168
Hi @dongjoon-hyun I try the test tens of times and the test failed twice. As we discussed with @viirya and @cloud-fan in https://github.com/apache/spark/pull/30998, the solution is not perfect. The order of calling materialize can guarantee that the order of task to be scheduled in normal circumstances, but, the guarantee is not strict since the submit of broadcast job and shuffle map job are in different thread. So there's still risk for the shuffle map job schedule earlier before broadcast job. I wonder should we need to remove the UT until we thorough resolve the issue. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
