Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/6586#issuecomment-112051808
@JoshRosen
The netty direct buffer leak problem fix on Netty-4.0.29-Final, but it is
not release official yet.
Although I also not sure that patch is needed in common or not.
Because our PageRank app is always out of controller of DirectBuffer, I
just want it be more Controllable. when lost of control, it will kill by yarn,
and then cause retry. Yarn is strictly limited physical memory space. Yean, it
is can enlarge memoryOverHead, and then enlarge the physical memory resource
for that app, but that overhead memory is used just for block transfer phase,
which be idle for other time. So in yarn mode, Iâm tend to make that direct
buffer space be more controller instead sacrifice some performance to write or
read block through fileChannel.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
