[
https://issues.apache.org/jira/browse/YARN-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263174#comment-15263174
]
Sangjin Lee commented on YARN-3959:
-----------------------------------
{quote}
Are we posting configurations for all YARN applications, or we just do that for
MapReduce apps? Actually, if we have system level support to post configs for
all YARN applications, we do not need to change much on the MR side, right? I
think configs are very general for YARN apps, so maybe we can fix that on
YARN's level rather than MR level?
{quote}
I am not sure if there is a YARN-generic way of writing the configuration,
regardless of the frameworks. First of all, the notion of configuration is not
universal and its existence/format/data is up to the framework. For example,
distributed shell does not have its own configuration, MR has configuration
({{JobConf}}) which extends {{Configuration}}, and Spark has its own
independent configuration ({{SparkConf}}) which does not derive from
{{Configuration}}. Also, even if such a configuration existed, I'm not sure if
they are ever sent to the RM, etc. so it can be written out to the timeline
service in a single place. I'd be curious to hear your thoughts on possible
mechanisms.
If there is no generic way of doing this, I think we might want to re-word this
JIRA to make it specific to mapreduce. I'd be +1 on moving this JIRA to the
MAPREDUCE project.
Regarding handling the size, the following may be necessary:
- split writing the configuration into multiple writes
- limit the overall size of the configuration (beyond which keys/values will be
dropped?)
- limit the size of individual values (beyond which the said key/value will be
dropped/truncated?)
This needs a little bit of design consideration (cc [~jrottinghuis]
[~vrushalic] for the hRaven experience). As we discussed offline, IMO it is
acceptable to do a simple write for now but handle the large configuration
issue in a later JIRA. I'd like to hear what others think.
> Store application related configurations in Timeline Service v2
> ---------------------------------------------------------------
>
> Key: YARN-3959
> URL: https://issues.apache.org/jira/browse/YARN-3959
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Junping Du
> Assignee: Varun Saxena
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-3959-YARN-2928.01.patch
>
>
> We already have configuration field in HBase schema for application entity.
> We need to make sure AM write it out when it get launched.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
