Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16989
looks pretty good, one last comment: since the shuffle fetcher doesn't
support spilling to disk, we should be very careful to avoid allocating a lot
of memory. Normally this should be fine as the total used memory will be under
`maxBytesInFlight`, but for single large shuffle blocks, we will allocate
memory for it even its size is over `maxBytesInFlight`. How about we always
fetch to disk if the block size is over `maxBytesInFlight`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
