Github user ConeyLiu commented on the issue:
https://github.com/apache/spark/pull/19135
Hi @jerryshao thanks for your reviewing.
>So it somehow reflects that CPU core contention is the main issue for
memory pre-occupation
I have modified the code, now it will not request more memory, now it just
reduce the times of calling `reserveUnrollMemoryForThisTask` followed by
@cloud-fan comments. And also the method is same as `putIteratorAsValues`.
Yeah, its impact will be small with small cores. In the above test results,
it doesn't bring any regressions, and also better for many cores. For machine
learning, we need cache the source data to OFF_HEAP in order to reduce the gc
problem.
For the configuration, I think the different application scenarios may be
different.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
