[
https://issues.apache.org/jira/browse/YARN-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952795#comment-15952795
]
Roni Burd commented on YARN-6426:
---------------------------------
Agreed. The proposed patch would allow to read uncompressed data, so it is
backward compatible. I can also add some config param to turn it off/on.
For forward compatibility is a bit trickier. If a new version is deployed and a
customer wants to go back, there will be no code ability to read the GZIP
format, so the only way of properly doing it will have to be via config and the
users doing this:
1) deploying new code with compression off..
2) then turn it on
> Compress ZK YARN keys to scale up (especially AppStateData
> ----------------------------------------------------------
>
> Key: YARN-6426
> URL: https://issues.apache.org/jira/browse/YARN-6426
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 3.0.0-alpha2
> Reporter: Roni Burd
> Assignee: Roni Burd
> Labels: patch
> Attachments: zkcompression.patch
>
>
> ZK today stores the protobuf files uncompressed. This is not an issue except
> that if a customer job has thousands of files, AppStateData will store the
> user context as a string with multiple URLs and it is easy to get to 1MB or
> more.
> This can put unnecessary strain on ZK and make the process slow.
> The proposal is to simply compress protobufs before sending them to ZK
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
