I have a very simple program that runs fine on my Linux server that runs
Spark master and worker in standalone mode.

public class SimpleSpark {
    public int sum () {
        SparkConf conf = new SparkConf()
                .setAppName("Magellan")
                .setMaster("spark://
ec2-nnn-nnn-nnn-nnn.compute-1.amazonaws.com:11407")
                .setJars(new String[]
{"target/magellan-spark-1.0-SNAPSHOT.jar"});
        JavaSparkContext sc = new JavaSparkContext(conf);

        List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
        JavaRDD<Integer> distData = sc.parallelize(data);
        int total = distData.reduce(new SumFunc());
        return total;
   }

    public static class SumFunc implements Function2<Integer, Integer,
Integer> {
        public Integer call(Integer a, Integer b) {
            return a + b;
        }
    };

However, when I run the same driver from a Windows machine it outputs the
following message and never completes:
  16/01/11 20:51:11 WARN TaskSchedulerImpl: Initial job has not accepted
any resources; check your cluster UI to ensure that workers are registered
and have sufficient resources

I have checked the cluster UI and the job is marked as RUNNING (so it does
not appears to be waiting on a worker).  I do not see anything out of the
ordinary in the master and worker logs.

How do I debug a problem like this?
-Andrew

Reply via email to