Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/6679#issuecomment-109499640
There's a situation in which there could be a behavior change in situations
where the executor somehow has a different Hadoop configuration file than the
driver. But I think it's the right change. I started to explain this stuff
abstractly, but I think it might be easier to just put down some examples:
**Example 1**
core-site.xml on the driver contains optionA->value1
core-site.xml on the executor contains optionA->value2
Old behavior: on the executor, conf.get("optionA") returns value1
New behavior: same as old behavior
**Example 2**
core-site.xml on the driver does not contain optionA
core-site.xml on the executor contains optionA->value1
Old behavior: on the executor, conf.get("optionA") returns value1
New behavior: on the executor, conf.get("optionA") returns null
I can't find the JIRA, but I believe there was a recent change by @vanzin
that made it so that the executor would use a copy of the Hadoop configuration
files used on the driver. When that is the case, neither example 1 nor example
2 can occur.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
