Github user sryza commented on the pull request: https://github.com/apache/spark/pull/6679#issuecomment-109499640 There's a situation in which there could be a behavior change in situations where the executor somehow has a different Hadoop configuration file than the driver. But I think it's the right change. I started to explain this stuff abstractly, but I think it might be easier to just put down some examples: **Example 1** core-site.xml on the driver contains optionA->value1 core-site.xml on the executor contains optionA->value2 Old behavior: on the executor, conf.get("optionA") returns value1 New behavior: same as old behavior **Example 2** core-site.xml on the driver does not contain optionA core-site.xml on the executor contains optionA->value1 Old behavior: on the executor, conf.get("optionA") returns value1 New behavior: on the executor, conf.get("optionA") returns null I can't find the JIRA, but I believe there was a recent change by @vanzin that made it so that the executor would use a copy of the Hadoop configuration files used on the driver. When that is the case, neither example 1 nor example 2 can occur.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org