[
https://issues.apache.org/jira/browse/SPARK-20369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982489#comment-15982489
]
Hyukjin Kwon edited comment on SPARK-20369 at 4/25/17 7:24 AM:
---------------------------------------------------------------
It looks I can't reproduce this as below:
{code}
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName("spark-conf-test") \
.setMaster("local[2]") \
.set('spark.python.worker.memory',"1g") \
.set('spark.executor.memory',"3g") \
.set("spark.driver.maxResultSize","2g")
print
print "Spark Config values in SparkConf:"
print conf.toDebugString()
sc = SparkContext(conf=conf)
print
print "Actual Spark Config values:"
print sc.getConf().toDebugString()
print conf.get("spark.python.worker.memory") ==
sc.getConf().get("spark.python.worker.memory")
print conf.get("spark.executor.memory") ==
sc.getConf().get("spark.executor.memory")
print conf.get("spark.driver.maxResultSize") ==
sc.getConf().get("spark.driver.maxResultSize")
{code}
{code}
Spark Config values in SparkConf:
spark.master=local[2]
spark.executor.memory=3g
spark.python.worker.memory=1g
spark.app.name=spark-conf-test
spark.driver.maxResultSize=2g
...
Actual Spark Config values:
...
spark.driver.maxResultSize=2g
spark.app.name=spark-conf-test
spark.executor.memory=3g
spark.master=local[2]
spark.python.worker.memory=1g
...
True
True
True
{code}
Are you able to check this in the current master maybe?
was (Author: hyukjin.kwon):
It looks I can't reproduce this as below:
{code}
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName("spark-conf-test") \
.setMaster("local[2]") \
.set('spark.python.worker.memory',"1g") \
.set('spark.executor.memory',"3g") \
.set("spark.driver.maxResultSize","2g")
print
print "Spark Config values in SparkConf:"
print conf.toDebugString()
sc = SparkContext(conf=conf)
print
print "Actual Spark Config values:"
print sc.getConf().toDebugString()
print conf.get("spark.python.worker.memory") ==
sc.getConf().get("spark.python.worker.memory")
print conf.get("spark.executor.memory") ==
sc.getConf().get("spark.executor.memory")
print conf.get("spark.driver.maxResultSize") ==
sc.getConf().get("spark.driver.maxResultSize")
{code}
{code}
Spark Config values in SparkConf:
spark.master=local[2]
spark.executor.memory=3g
spark.python.worker.memory=1g
spark.app.name=spark-conf-test
spark.driver.maxResultSize=2g
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
17/04/25 16:20:08 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Actual Spark Config values:
spark.app.id=local-1493104809510
spark.app.name=spark-conf-test
spark.driver.extraClassPath=/Users/hyukjinkwon/Desktop/workspace/local/forked/spark-xml/target/scala-2.11/spark-xml_2.11-0.4.0.jar
spark.driver.host=192.168.15.168
spark.driver.maxResultSize=2g
spark.driver.port=56783
spark.executor.extraClassPath=/Users/hyukjinkwon/Desktop/workspace/local/forked/spark-xml/target/scala-2.11/spark-xml_2.11-0.4.0.jar
spark.executor.id=driver
spark.executor.memory=3g
spark.master=local[2]
spark.python.worker.memory=1g
spark.rdd.compress=True
spark.serializer.objectStreamReset=100
spark.submit.deployMode=client
True
True
True
{code}
Are you able to check this in the current master maybe?
> pyspark: Dynamic configuration with SparkConf does not work
> -----------------------------------------------------------
>
> Key: SPARK-20369
> URL: https://issues.apache.org/jira/browse/SPARK-20369
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.1.0
> Environment: Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-40-generic x86_64)
> and Mac OS X 10.11.6
> Reporter: Matthew McClain
> Priority: Minor
>
> Setting spark properties dynamically in pyspark using SparkConf object does
> not work. Here is the code that shows the bug:
> ---
> from pyspark import SparkContext, SparkConf
> def main():
> conf = SparkConf().setAppName("spark-conf-test") \
> .setMaster("local[2]") \
> .set('spark.python.worker.memory',"1g") \
> .set('spark.executor.memory',"3g") \
> .set("spark.driver.maxResultSize","2g")
> print "Spark Config values in SparkConf:"
> print conf.toDebugString()
> sc = SparkContext(conf=conf)
> print "Actual Spark Config values:"
> print sc.getConf().toDebugString()
> if __name__ == "__main__":
> main()
> ---
> Here is the output; none of the config values set in SparkConf are used in
> the SparkContext configuration:
> Spark Config values in SparkConf:
> spark.master=local[2]
> spark.executor.memory=3g
> spark.python.worker.memory=1g
> spark.app.name=spark-conf-test
> spark.driver.maxResultSize=2g
> 17/04/18 10:21:24 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Actual Spark Config values:
> spark.app.id=local-1492528885708
> spark.app.name=sandbox.py
> spark.driver.host=10.201.26.172
> spark.driver.maxResultSize=4g
> spark.driver.port=54657
> spark.executor.id=driver
> spark.files=file:/Users/matt.mcclain/dev/datascience-experiments/mmcclain/client_clusters/sandbox.py
> spark.master=local[*]
> spark.rdd.compress=True
> spark.serializer.objectStreamReset=100
> spark.submit.deployMode=client
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
