Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18388
I don't think we reject any requests at this point. So yes you could still
run into an issue. Generally I think limiting the # of blocks you are fetching
at once will solve this also, but its not guaranteed. By default we get one
chunk per block, so again I think the most requests you would have buffered
would be spark.reducer.maxBlocksInFlightPerAddress * # reducer connected.
I'm still ok with adding in something like what you have here as a last
resort type thing. I recommend just closing the connections rather then
changing the api to keep backwards compatibility. If there is some issue with
that though then perhaps we can config it off by default like you suggest but I
think that is much harder for the broader community to see benefit from it.
You mentioned a problem with just doing a close, what was the problem?
The flow control part I mention above is really just if the outbound
buffers are taking to much memory. I think doing all 3 of these would be good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
