Hello community,
I have a question about container resource calculation in nodemanager.
Some time ago a filed JIRA
https://issues.apache.org/jira/browse/YARN-4681, which I though might
address our problems with container being killed because of read-only
mmaping memory block. The JIRA has not been resolved yet, but it turned
out for us, that the patch doesn't solve the problem. Some applications
(namely Apache Spark) tend to allocate really large memory blocks
outside JVM heap (using mmap, but with MAP_PRIVATE), but only for short
time periods. We solved this by creating a smoothing resource
calculator, which averages the memory usage of a container over some
time period (say 5 minutes). This eliminates the problem of container
being killed for short memory consumption peak, but in the same time
preserves the ability to kill container that *really* consumes excessive
amount of memory.
My question is, does this seem a systematic approach to you and should I
post our patch to the community or am thinking in a wrong direction from
the beginning? :)
Thanks for reactions,
Jan
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
