Roni Burd created YARN-6426:
-------------------------------
Summary: Compress ZK YARN keys to scale up (especially AppStateData
Key: YARN-6426
URL: https://issues.apache.org/jira/browse/YARN-6426
Project: Hadoop YARN
Issue Type: Improvement
Components: yarn
Reporter: Roni Burd
Assignee: Roni Burd
Fix For: 2.8.0, 3.0.0-alpha2
ZK today stores the protobuf files uncompressed. This is not an issue except
that if a customer job has thousands of files, AppStateData will store the user
context as a string with multiple URLs and it is easy to get to 1MB or more.
This can put unnecessary strain on ZK and make the process slow.
The proposal is to simply compress protobufs before sending them to ZK
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
