otterc commented on PR #36411: URL: https://github.com/apache/spark/pull/36411#issuecomment-1115410092
@wankunde Can you please provide more details/logs of the problem that you are trying to solve. In specific, can you provide some logs that exhibit the below > After the shuffle stage is finalized, the reduce fetch blocks RPC will be blocked if there are still many map output blocks in flight. `ShuffleBlockPusher` already uses config `maxBlocksInFlightPerAddress`. Why do we need a new configuration for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
