[ https://issues.apache.org/jira/browse/SPARK-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Egor Pahomov closed SPARK-4403. ------------------------------- Resolution: Invalid > Elastic allocation(spark.dynamicAllocation.enabled) results in task never > being execued. > ---------------------------------------------------------------------------------------- > > Key: SPARK-4403 > URL: https://issues.apache.org/jira/browse/SPARK-4403 > Project: Spark > Issue Type: Bug > Components: Spark Core, YARN > Affects Versions: 1.1.1 > Reporter: Egor Pahomov > Attachments: ipython_out > > > I execute ipython notebook + pyspark with spark.dynamicAllocation.enabled = > true. Task never ends. > Code: > {code} > import sys > from random import random > from operator import add > partitions = 10 > n = 100000 * partitions > def f(_): > x = random() * 2 - 1 > y = random() * 2 - 1 > return 1 if x ** 2 + y ** 2 < 1 else 0 > count = sc.parallelize(xrange(1, n + 1), partitions).map(f).reduce(add) > print "Pi is roughly %f" % (4.0 * count / n) > {code} > {code} > IPYTHON_ARGS="notebook --profile=ydf --port $IPYTHON_PORT --port-retries=0 > --ip='*' --no-browser" > pyspark \ > --verbose \ > --master yarn-client \ > --conf spark.driver.port=$((RANDOM_PORT + 2)) \ > --conf spark.broadcast.port=$((RANDOM_PORT + 3)) \ > --conf spark.replClassServer.port=$((RANDOM_PORT + 4)) \ > --conf spark.blockManager.port=$((RANDOM_PORT + 5)) \ > --conf spark.executor.port=$((RANDOM_PORT + 6)) \ > --conf spark.fileserver.port=$((RANDOM_PORT + 7)) \ > --conf spark.shuffle.service.enabled=true \ > --conf spark.dynamicAllocation.enabled=true \ > --conf spark.dynamicAllocation.minExecutors=1 \ > --conf spark.dynamicAllocation.maxExecutors=10 \ > --conf spark.ui.port=$SPARK_UI_PORT > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org