Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/14239
thanks for the explanation, this makes much more sense now. I'm still a bit
concerned about the memory usage of this though, especially with external
shuffle on the nodemanager.
Were you using the external shuffle to test this or just the shuffle built
into the executors? How much memory did you give whatever was shuffling and
how big were the blocks being fetched?
Does this look at all about the size its trying to cache vs size available
to shuffle handler?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
