Hey Egor, Have you checked the AM logs? My guess is that it threw an exception or something such that no executors (not even the initial set) have registered with your driver. You may already know this, but you can go to the http://<RM address>:8088 page and click into the application to access this. Alternatively you could run "yarn logs -applicationId <appId>" after quitting the application.
-Andrew 2014-11-14 14:23 GMT-08:00 Sandy Ryza <sandy.r...@cloudera.com>: > That would be helpful as well. Can you confirm that when you try it with > dynamic allocation the cluster has free resources? > > On Fri, Nov 14, 2014 at 12:17 PM, Egor Pahomov <pahomov.e...@gmail.com> > wrote: > >> It's successful without dynamic allocation. I can provide spark log for >> that scenario if it can help. >> >> 2014-11-14 21:36 GMT+02:00 Sandy Ryza <sandy.r...@cloudera.com>: >> >>> Hi Egor, >>> >>> Is it successful without dynamic allocation? From your log, it looks >>> like the job is unable to acquire resources from YARN, which could be >>> because other jobs are using up all the resources. >>> >>> -Sandy >>> >>> On Fri, Nov 14, 2014 at 11:32 AM, Egor Pahomov <pahomov.e...@gmail.com> >>> wrote: >>> >>>> Hi. >>>> I execute ipython notebook + pyspark with >>>> spark.dynamicAllocation.enabled = true. Task never ends. >>>> Code: >>>> >>>> import sys >>>> from random import random >>>> from operator import add >>>> partitions = 10 >>>> n = 100000 * partitions >>>> >>>> def f(_): >>>> x = random() * 2 - 1 >>>> y = random() * 2 - 1 >>>> return 1 if x ** 2 + y ** 2 < 1 else 0 >>>> >>>> count = sc.parallelize(xrange(1, n + 1), partitions).map(f).reduce(add) >>>> print "Pi is roughly %f" % (4.0 * count / n) >>>> >>>> >>>> >>>> Run notebook: >>>> >>>> IPYTHON_ARGS="notebook --profile=ydf --port $IPYTHON_PORT --port-retries=0 >>>> --ip='*' --no-browser" >>>> pyspark \ >>>> --verbose \ >>>> --master yarn-client \ >>>> --conf spark.driver.port=$((RANDOM_PORT + 2)) \ >>>> --conf spark.broadcast.port=$((RANDOM_PORT + 3)) \ >>>> --conf spark.replClassServer.port=$((RANDOM_PORT + 4)) \ >>>> --conf spark.blockManager.port=$((RANDOM_PORT + 5)) \ >>>> --conf spark.executor.port=$((RANDOM_PORT + 6)) \ >>>> --conf spark.fileserver.port=$((RANDOM_PORT + 7)) \ >>>> --conf spark.shuffle.service.enabled=true \ >>>> --conf spark.dynamicAllocation.enabled=true \ >>>> --conf spark.dynamicAllocation.minExecutors=1 \ >>>> --conf spark.dynamicAllocation.maxExecutors=10 \ >>>> --conf spark.ui.port=$SPARK_UI_PORT >>>> >>>> >>>> Spark/Ipython log is in attachment. >>>> >>>> -- >>>> >>>> >>>> >>>> *Sincerely yoursEgor PakhomovScala Developer, Yandex* >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>> >>> >> >> >> -- >> >> >> >> *Sincerely yoursEgor PakhomovScala Developer, Yandex* >> > >
