Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1197#issuecomment-47140832
I'd argue that (b) is also an urgent issue.
Yes, for the particular case of not being able to talk to the master, an
exception is thrown (though after a long timeout). Now consider the case in
which you can talk to the master, but there are no workers. Then in the normal
case, a simple Spark job leads to the following:
```
>>> sc.parallelize(range(100)).count()
14/06/25 11:26:12 INFO SparkContext: Starting job: count at <stdin>:1
14/06/25 11:26:12 INFO DAGScheduler: Got job 0 (count at <stdin>:1) with 2
output partitions (allowLocal=false)
14/06/25 11:26:12 INFO DAGScheduler: Final stage: Stage 0(count at
<stdin>:1)
14/06/25 11:26:12 INFO DAGScheduler: Parents of final stage: List()
14/06/25 11:26:12 INFO DAGScheduler: Missing parents: List()
14/06/25 11:26:12 INFO DAGScheduler: Submitting Stage 0 (PythonRDD[1] at
RDD at PythonRDD.scala:40), which has no missing parents
14/06/25 11:26:12 INFO DAGScheduler: Submitting 2 missing tasks from Stage
0 (PythonRDD[1] at RDD at PythonRDD.scala:40)
14/06/25 11:26:12 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
14/06/25 11:26:27 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient memory
14/06/25 11:26:42 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient memory
14/06/25 11:26:57 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient memory
...
```
Now imagine all the WARN messages go away. This will continue indefinitely
and is basically a hang. I think this is more serious than a user experience
issue and deserves a hot fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
