[
https://issues.apache.org/jira/browse/SPARK-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339498#comment-14339498
]
Apache Spark commented on SPARK-6048:
-------------------------------------
User 'andrewor14' has created a pull request for this issue:
https://github.com/apache/spark/pull/4799
> SparkConf.translateConfKey should translate on get, not set
> -----------------------------------------------------------
>
> Key: SPARK-6048
> URL: https://issues.apache.org/jira/browse/SPARK-6048
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Critical
>
> There are several issues with translating on set.
> (1) The most serious one is that if the user has both the deprecated and the
> latest version of the same config set, then the value picked up by SparkConf
> will be arbitrary. Why? Because during initialization of the conf we call
> `conf.set` on each property in `sys.props` in an order arbitrarily defined by
> Java. Instead, we should always use the value of the latest version of the
> config if that is provided.
> (2) If we translate on set, then we must keep translating everywhere else. In
> fact, the current code does not translate on remove, which means the
> following won't work if X is deprecated:
> {code}
> conf.set(X, Y)
> conf.remove(X) // X is not in the conf
> {code}
> (3) Since we call `conf.set` on all configs when initializing the conf, we
> print all deprecation warnings in the beginning. Elsewhere in Spark, however,
> we warn the user when the deprecated config / option / env var is actually
> being used.
> We should keep this consistent so the user won't expect to find all
> deprecation messages in the beginning of his logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
