[
https://issues.apache.org/jira/browse/HADOOP-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12571294#action_12571294
]
Aaron Kimball commented on HADOOP-2866:
---------------------------------------
@Doug: Adding a warning on hadoop-site defining a value absent from
hadoop-defaults is a good idea.
@Owen: If we performed a major refactoring of command names, we could perhaps
add a very small checklist registry in JobConf, which only examines the
top-level component of the key being set. So if we move all of mapred.\* into
hadoop.mapred.\*, for example, the top-level namespaces "mapred", "fs", "dfs",
"io", etc. could be made to trigger a warning that the key has been deprecated
(and the subsequent release could fail with an error that it is now no-longer
used); setting values inside "hadoop.\*" is definitely ok. Setting values
outside this family of namespaces entirely (user-defined keys) also would
provoke no warning.
So what particular actions do we want to take?
Any/all of the following?
1) Add accessor/setter methods for all entries to JobConf and refactor all
classes to use them (yikes)
2) Move all key names into hadoop.\*
2a) Determine more sane and consistent names for the key names as long as we're
moving them? (this requires a definition of what constitutes 2nd- and 3rd-level
namespaces)
3) Add a top-level key namespace deprecation warning based on hard-coded list
of deprecated top-level namespaces
4) divorce set(k, v) from setHadoop(k, v); only setHadoop() can set values in
hadoop.\*; it can be private to JobConf, which will force the use of
JobConf-blessed method names (note: in the first release, set() will allow
setting of hadoop.\*, but will print a deprecation warning; the next release
will ban this behavior.)
5) Something else..?
> JobConf should validate key names in well-defined namespaces and warn on
> misspelling
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-2866
> URL: https://issues.apache.org/jira/browse/HADOOP-2866
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Aaron Kimball
> Priority: Minor
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> A discussion on the mailing list reveals that some configuration strings in
> the JobConf are deprecated over time and new configuration names replace them:
> e.g., "mapred.output.compression.type" is now replaced with
> "mapred.map.output.compression.type"
> Programmers who have been manually specifying the former string, however,
> receive no diagnostic output during testing to suggest that their compression
> type is being silently ignored.
> It would be desirable to notify developers of this change by printing a
> warning message when deprecated configuration names are used in a newer
> version of Hadoop. More generally, when any configuration string in the
> mapred.\*, fs.\*, dfs.\*, etc namespaces are provided by a user and are not
> recognized by Hadoop, it is desirable to print a warning, to indicate
> malformed configurations. No warnings should be printed when configuration
> keys are in user-defined namespaces (e.g., "myprogram.mytask.myvalue").
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
