Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18388
OK then let's go with the flow control direction.
> For the flow control part, I'm just worrying the queue will be too large
and causing memory issue.
We can make an external queue, i.e. if it's too large, spill to disk.
Another concern is, with flow control, shuffle service may hang a request
for a long time, and cause the client to timeout and fail. It's better than
just closing the connection, but there is still a chance that the client
mistakenly reports FetchFailure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
