Manuel Godbert created TEZ-3461:
-----------------------------------
Summary: Tez not working in local mode for M/R jobs
Key: TEZ-3461
URL: https://issues.apache.org/jira/browse/TEZ-3461
Project: Apache Tez
Issue Type: Bug
Reporter: Manuel Godbert
I have map/reduce jobs that work as expected within YARN, and I want to see if
Tez can help me improving their performance. Alas, I am experiencing issues and
I want to understand what happens, to see if I can adapt my code or if I can
suggest Tez enhancements. For this I need to be able to debug jobs from within
eclipse, with breakpoints in Tez source code etc.
I am working on a linux (ubuntu) platform
I use the latest Tez version I found, i.e. 0.9.0-SNAPSHOT (also tried with
0.7.0)
I have set up the hortonworks mini dev cluster
https://github.com/hortonworks/mini-dev-cluster
I am trying to run the basic WordCount2 code found here
https://hadoop.apache.org/docs/r2.7.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v2.0
I added the following code to have tez running locally:
conf.set("mapreduce.framework.name", "yarn-tez");
conf.setBoolean("tez.local.mode", true);
conf.set("fs.default.name", "file:///");
conf.setBoolean("tez.runtime.optimize.local.fetch", true);
And I am getting the following error:
2016-09-27 18:32:34 Running Dag: dag_1474992804027_0003_1
2016-09-27 18:32:34 Running Dag: dag_1474992804027_0003_1
Exception in thread "main" java.lang.NullPointerException
at
org.apache.tez.client.LocalClient.getApplicationReport(LocalClient.java:153)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getAppReport(DAGClientRPCImpl.java:231)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.createAMProxyIfNeeded(DAGClientRPCImpl.java:251)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:96)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:360)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:220)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:268)
at
org.apache.tez.dag.api.client.MRDAGClient.getDAGStatus(MRDAGClient.java:58)
at
org.apache.tez.mapreduce.client.YARNRunner.getJobStatus(YARNRunner.java:710)
at
org.apache.tez.mapreduce.client.YARNRunner.submitJob(YARNRunner.java:650)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
at WordCount2.main(WordCount2.java:136)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
