[
https://issues.apache.org/jira/browse/SPARK-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhang updated SPARK-11099:
-------------------------------
Description:
spark.driver.extraClassPath doesn't take effect in the latest code, and find
the root cause is due to the default conf property file is not loaded
The bug is caused by this code snippet in AbstractCommandBuilder
{code}
Map<String, String> getEffectiveConfig() throws IOException {
if (effectiveConfig == null) {
if (propertiesFile == null) {
effectiveConfig = conf; // return from here if no propertyFile is
provided
} else {
effectiveConfig = new HashMap<>(conf);
Properties p = loadPropertiesFile(); // default propertyFile
will load here
for (String key : p.stringPropertyNames()) {
if (!effectiveConfig.containsKey(key)) {
effectiveConfig.put(key, p.getProperty(key));
}
}
}
}
return effectiveConfig;
}
{code}
was:
spark.driver.extraClassPath doesn't take effect in the latest code, and find
the root cause is due to the default conf property file is not loaded
The bug is caused by this code snippet in AbstractCommandBuilder
{code}
Map<String, String> getEffectiveConfig() throws IOException {
if (effectiveConfig == null) {
if (propertiesFile == null) {
effectiveConfig = conf;
} else {
effectiveConfig = new HashMap<>(conf);
Properties p = loadPropertiesFile();
for (String key : p.stringPropertyNames()) {
if (!effectiveConfig.containsKey(key)) {
effectiveConfig.put(key, p.getProperty(key));
}
}
}
}
return effectiveConfig;
}
{code}
> Default conf property file is not loaded
> -----------------------------------------
>
> Key: SPARK-11099
> URL: https://issues.apache.org/jira/browse/SPARK-11099
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell, Spark Submit
> Reporter: Jeff Zhang
> Priority: Critical
>
> spark.driver.extraClassPath doesn't take effect in the latest code, and find
> the root cause is due to the default conf property file is not loaded
> The bug is caused by this code snippet in AbstractCommandBuilder
> {code}
> Map<String, String> getEffectiveConfig() throws IOException {
> if (effectiveConfig == null) {
> if (propertiesFile == null) {
> effectiveConfig = conf; // return from here if no propertyFile
> is provided
> } else {
> effectiveConfig = new HashMap<>(conf);
> Properties p = loadPropertiesFile(); // default propertyFile
> will load here
> for (String key : p.stringPropertyNames()) {
> if (!effectiveConfig.containsKey(key)) {
> effectiveConfig.put(key, p.getProperty(key));
> }
> }
> }
> }
> return effectiveConfig;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
