https://issues.apache.org/jira/browse/YARN-321
There is not a generic application history server yet. The current one works for MR. On Tue, Dec 9, 2014 at 4:48 PM, Nagy István <[email protected]> wrote: > Dear all, > > I would like to run a simple spark job on EMR with yarn. > > My job is the follows: > > public void EMRRun() { > SparkConf sparkConf = new > SparkConf().setAppName("RunEMR").setMaster("yarn-cluster"); > sparkConf.set("spark.executor.memory", "13000m"); JavaSparkContext ctx = > new JavaSparkContext(sparkConf); System.out.println(ctx.appName()); > List<Integer> list = new LinkedList<Integer>(); for (int i =0; i<10000; > i++){ > list.add(i); } > > JavaRDD<Integer> listRDD = ctx.parallelize(list); List<Integer> > results = listRDD.collect(); for (Integer i : results){ > System.out.println(i); } > > ctx.stop();} > public static void main(String[] args) { > SparkTest sp = new SparkTest(); sp.EMRRun();} > > > On EMR I run the spark with spark-submit with the following: > > ./spark-submit --class > com.collokia.ml.stackoverflow.usertags.browserhistory.sparkTestJava.SparkTest > --master yarn-cluster --executor-memory 512m --num-executors 10 > /home/hadoop/MLyBigData.jar > > After that finished I tried to see yarn log, but I got this: > yarn logs -applicationId application_1418123020170_0032 > 14/12/09 20:29:26 INFO client.RMProxy: Connecting to ResourceManager at / > 172.31.3.155:9022 > Logs not available at /tmp/logs/hadoop/logs/application_1418123020170_0032 > Log aggregation has not completed or is not enabled. > > But I modified the yarn-site.xml as: > > <property><name>yarn.log-aggregation-enable</name><value>true</value></property> > > <property><name>yarn.log-aggregation.retain-seconds</name><value>-1</value></property> > > <property><name>yarn.log-aggregation.retain-check-interval-seconds</name><value>30</value></property> > > I use AMI version of 3.2.3, spark 1.1.0 on hadoop 2.4 > > Any suggestions how can I see the logs of the yarn? > Thanks, > Istvan > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
