Wangda Tan created MAPREDUCE-7309:
-------------------------------------
Summary: Improve performance of reading resource request for
mapper/reducers from config
Key: MAPREDUCE-7309
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7309
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: applicationmaster
Affects Versions: 3.3.0, 3.2.0, 3.1.0, 3.0.0
Reporter: Wangda Tan
Attachments: MAPREDUCE-7309.001.patch
This is an issue could affect all the releases which includes YARN-6927.
Basically, we use regex match repeatly when we read mapper/reducer resource
request from config files. When we have large config file, and large number of
splits, it could take a long time.
We saw AM could take hours to parse config when we have 200k+ splits, with a
large config file (hundreds of kbs).
We should do proper caching for pre-configured resource requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
