[
https://issues.apache.org/jira/browse/HADOOP-785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12519481
]
Arun C Murthy commented on HADOOP-785:
--------------------------------------
I'd like to assuage some concerns and clarify/change some more about certain
aspects of the above proposal, here we go...
Existing issues:
a) Easy to misconfigure (e.g. Owen's original description) and hard to debug.
b) No way to separately configure the servers and clients (both dfs & mapred
clients).
c) Ease administration of large clusters, ensuring that administrators have a
fair amount of control over some configuration knobs, while at the same time
ensuring users can control *relevant" config knobs.
d) No way to override hadoop-site.xml from job.xml (e.g. turn on/off
speculative execution from job.xml).
*Proposal*:
Organise the config tree around where things in cluster are used i.e. things
used only by server, and knobs clients (both hdfs and mapred clients) may tweak.
To help clarify the above I propose we re-work the config files as follows:
a) hadoop-default.xml - Read-only defaults.
b) hadoop-site.xml - General site-specific values e.g. you'd put
{{fs.default.name}} and {{mapred.job.tracker}} here. Important change: values
here *can be overridden* (code-speak: we'd do
{{addDefaultResource(hadoop-site)}} rather than
{{addFinalResource(hadoop-site)}} as done today).
c) hadoop-server.xml - Hadoop servers (daemons) are configured via this, and
values here *cannot* be overridden. One would, for e.g., put
{{dfs.replication}} here.
d) hadoop-client.xml - Hadoop clients (bot hdfs and mapred clients) are
configured via this file, which, again, cannot be overridden. One would, for
e.g., put {{dfs.client.buffer.dir}} here.
e) job.xml - Job-specific configuration at run-time (as today).
Thus, hadoop-site.xml, which is the non-overridable today, is kept around as a
general, site-specific, overridable configuration file, where *any* config knob
can be tweaked. We could completely do away with hadoop-site.xml, however to
keep things backward compatible and to ensure that common configuration isn't
duplicated (else, we would have to force users to specify
{{fs.default.name}}/{{mapred.job.tracker}} in both hadoop-server.xml &
hadoop-client.xml, which seems redundant).
However, hadoop-server.xml and hadoop-client.xml are 2 new config files for
hadoop servers (daemons) and clients which *cannot* be overridden in the server
and client context respectively.
The class heirarchy now looks like this:
{noformat}
abstract public class Configuration {
// ...
}
public class ServerConfiguration extends Configuration {
// reads hadoop-default.xml, hadoop-site.xml & hadoop-server.xml, in that
order
}
public class ClientConfiguration extends Configuration {
// reads hadoop-default.xml, hadoop-site.xml & hadoop-client.xml, in that
order
}
public class JobConf extends ClientConfiguration {
// reads hadoop-default.xml, hadoop-site.xml, hadoop-client.xml & job.xml
in that order
}
{noformat}
I propose we tighten interfaces of servers/clients to reflect that they are
using a specific *type* of configuration. E.g.
{noformat}
class JobTracker {
public JobTracker(JobConf conf) {
// ...
}
}
{noformat}
becomes:
{noformat}
class JobTracker {
public JobTracker(ServerConfiguration conf) {
// ...
}
}
{noformat}
and
{noformat}
public class DFSClient {
public DFSClient(Configuration conf) {
// ...
}
}
{noformat}
becomes:
{noformat}
public class DFSClient {
public DFSClient(ClientConfiguration conf) {
// ...
}
}
{noformat}
and so on... (Map/Reduce public apis already correctly defined in terms of
JobConf already).
Thus, we would create different types of configuration objects
(ServerConfiguration/ClientConfiguration/JobConf) and use them in the relevant
sub-systems (NameNode/JobTracker: ServerConfiguration, DFSClient/MapReduceBase:
ClientConfiguration/JobConf) etc.
This has the benefit that
a) It matches the context and use cases of the designated configuration files.
b) Users have a fair amount of control over *relevant* knobs e.g. they can
specify {{io.buffer.size}} in JobConf and that will be used by, for e.g., in
SequenceFile via {{InputFormat.getRecordReader(InputSplit, JobConf, Reporter)}}.
c) Ensures that administration of the large cluster is eased and that admins
have a fair amount of control over the various configuration knobs. E.g. there
is no way for a *user* to tweak {{dfs.client.buffer.dir}} via the JobConf since
DFSClient is defined in terms of ClientConfiguration which clearly never looks
at job.xml.
Also, I'd like to use this redo to go ahead and implement Doug's suggestion of
using static methods, where applicable, to setup the actual values; thereby
stop dumping too many get/set methods in Configuration/JobConf.
Last, not least, is how we go about communicating *where* the right place for
each config knob is; in this regard I guess the consensus has been to not
introduce newer xml tags/attributes and hence to use the *description* tag and
place it along with the 'level' (expert/intermediate etc.).
> Divide the server and client configurations
> -------------------------------------------
>
> Key: HADOOP-785
> URL: https://issues.apache.org/jira/browse/HADOOP-785
> Project: Hadoop
> Issue Type: Improvement
> Components: conf
> Affects Versions: 0.9.0
> Reporter: Owen O'Malley
> Assignee: Arun C Murthy
> Fix For: 0.15.0
>
>
> The configuration system is easy to misconfigure and I think we need to
> strongly divide the server from client configs.
> An example of the problem was a configuration where the task tracker has a
> hadoop-site.xml that set mapred.reduce.tasks to 1. Therefore, the job tracker
> had the right number of reduces, but the map task thought there was a single
> reduce. This lead to a hard to find diagnose failure.
> Therefore, I propose separating out the configuration types as:
> class Configuration;
> // reads site-default.xml, hadoop-default.xml
> class ServerConf extends Configuration;
> // reads hadoop-server.xml, $super
> class DfsServerConf extends ServerConf;
> // reads dfs-server.xml, $super
> class MapRedServerConf extends ServerConf;
> // reads mapred-server.xml, $super
> class ClientConf extends Configuration;
> // reads hadoop-client.xml, $super
> class JobConf extends ClientConf;
> // reads job.xml, $super
> Note in particular, that nothing corresponds to hadoop-site.xml, which
> overrides both client and server configs. Furthermore, the properties from
> the *-default.xml files should never be saved into the job.xml.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
