[GitHub] [spark] zhongyu09 commented on pull request #31099: [SPARK-33933][SQL] Remove UT before we completely fix SPARK-33933
zhongyu09 commented on pull request #31099: URL: https://github.com/apache/spark/pull/31099#issuecomment-757987871 > > some situations > > Mind elabourating what are the "some situations"? Because the submit of broadcast job and shuffle map job are in different thread, 1. for broadcast job, call doPrepare() in main thread, and then start the real materialization in "broadcast-exchange" thread pool: calling getByteArrayRdd().collect() to submit collect job 2. for shuffle map job, call ShuffleExchangeExec.mapOutputStatisticsFuture() which call sparkContext.submitMapStage() directly in main thread to submit map stage #1 is trigger in #2, so in normal cases, the broadcast job will be submit first. However, we can not control how fast the two thread runs, so the "broadcast-exchange" thread could run a little bit slower than main thread, result in map stage submit first This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhongyu09 commented on pull request #31099: [SPARK-33933][SQL] Remove UT before we completely fix SPARK-33933
zhongyu09 commented on pull request #31099: URL: https://github.com/apache/spark/pull/31099#issuecomment-757102568 @cloud-fan @viirya @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhongyu09 commented on pull request #31099: [SPARK-33933][SQL] Remove UT before we completely fix SPARK-33933
zhongyu09 commented on pull request #31099: URL: https://github.com/apache/spark/pull/31099#issuecomment-757102568 @cloud-fan @viirya @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
