[
https://issues.apache.org/jira/browse/YARN-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13801062#comment-13801062
]
Zhijie Shen commented on YARN-975:
----------------------------------
As to the scalability of the implementation, it's obvious that 1 application
will result in 1 history file.
Did some local test of 1 application, 1 application attempt and multiple
containers. The sizes of the file area:
| \# of containers | 100 | 1K | 10K | 100K |
| size of the file | 19KB | 184KB | 1.8MB | 19MB |
The size of the file is proportional to the number of containers. Even there're
100K containers of one the application, the file size seems to be still
acceptable. Of course, the file size is also determined by the variable-length
fields. For example, if every history data contains a pretty long logURL, the
file size will be certainly larger. However, the size should be on the same
order of magnitude.
> Add a file-system implementation for history-storage
> ----------------------------------------------------
>
> Key: YARN-975
> URL: https://issues.apache.org/jira/browse/YARN-975
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Zhijie Shen
> Assignee: Zhijie Shen
> Attachments: YARN-975.1.patch, YARN-975.2.patch, YARN-975.3.patch,
> YARN-975.4.patch, YARN-975.5.patch, YARN-975.6.patch
>
>
> HDFS implementation should be a standard persistence strategy of history
> storage
--
This message was sent by Atlassian JIRA
(v6.1#6144)
