KWON BYUNGCHANG created YARN-11960:
--------------------------------------
Summary: Fix NodeManager OOM caused by large client.json
Key: YARN-11960
URL: https://issues.apache.org/jira/browse/YARN-11960
Project: Hadoop YARN
Issue Type: Bug
Reporter: KWON BYUNGCHANG
In Spark or MapReduce jobs, when accessing private Docker registry, Docker
config.json path can be set by below environment variable:
{noformat}
YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=hdfs://ns/user/foo/config.json
{noformat}
NodeManager reads this file and writes to local filesystem as config.json. But
current implementation reads whole file to memory and do JSON parsing, then
writes it.
If large file is configured, NodeManager can occur OOM.
So file size limit should be added.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
