xuanyuanking commented on issue #26663: [SPARK-30015][Core] Continuous shuffle block fetching should be disabled by default when the old fetch protocol is used URL: https://github.com/apache/spark/pull/26663#issuecomment-558275507 > how things can go wrong if we do batch fetch with old shuffle protocol? That's the scenario Yuming met at https://github.com/apache/spark/pull/26147/files#r348943508. For the old shuffle service user, only `spark.shuffle.useOldFetchProtocol` set to true can job run. But if we batch fetch here, only shuffle block fetcher will merge the request because, in the old code, we do the continuous fetching without checking old fetch protocol. While the server-side would use the old protocol and use OpenBlocks in OneForOneBlockFetcher, which still fetches blocks one by one. So the size number between fetcher and iterator will be inconsistent.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
