[
https://issues.apache.org/jira/browse/YARN-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670153#comment-13670153
]
Siddharth Seth commented on YARN-702:
-------------------------------------
bq. The issue in hbase is that we'd like to have one single conf that gets the
new settings but the minicluster creates a new copy of the conf passed in
instead of augmenting it. Thus when hbase MR jobs run, we need to copy out a
whole bunch of settings back out.
There's some discussion about this on HBASE-7904.
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13611311&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611311
In the steps mentioned, is the same config object used ? Since MR is the last
step, is it possible to replace this configuration with the conf returned by
the MiniMRCluster (getConfiguration()).
If the MiniMRCluster were to use the conf object passed in as a parameter, it
would end up adding a bunch of keys to this config (Everything defined in
yarn-default and yarn-site) - effectively the same as a getConfiguration on the
cluster.
I think it's useful for the cluster to run with a different config, and not
share this with the multiple jobs that may be submitted to it. What may be
possible is for the MiniMRCluster to copy these configs into the user specified
config. That would be very similar to a blanket copy.
As mentioned before, I think it'll be useful to expose a limited configuration
set (RM address, isMiniCluster, etc) which can then be merged into client
configs. Alternately, have the MiniCluster merge in this limited set. This
could be a start towards separating cluster specific configuration and client
specific configuration.
> minicluster classpath construction requires user to set yarn.is.minicluster
> in the job conf
> -------------------------------------------------------------------------------------------
>
> Key: YARN-702
> URL: https://issues.apache.org/jira/browse/YARN-702
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.0.4-alpha
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
>
> YARN-129 improved classpath construction for miniclusters by, when
> yarn.is.minicluster is set, adding the current JVM's classpath to the
> ContainerLaunchContext for the MR AM and tasks. An issue with this is that
> it requires the user to set yarn.is.minicluster on the mapreduce side in the
> job conf, if they are not copying to RM conf into the jobconf.
> I think it would be better to bypass the ContainerLaunchContext and instead
> have the nodemanager check the property, and if it is true, do the classpath
> additions there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
