** appologies for sending multiple emails as the previous once got truncated some how ** Hi, I am struggling to find the root cause of pig jobs not running in Cluster Mode after we upgraded the cluster to use YARN (MRV2) from MRV1. When the job is submitted, I see the following message as the most obvious difference from the old cluster where the job was running fine on Cluster mode. :
On the new cluster (where Yarn is running) I see the following: It appears the job is not getting submitted to the RM. I have ensure the PIG_HOME and HADOOP_MAPRED_HOME is set right but despite of many tries this is what I am getting. Since the job admission isn't successful on YARN its failing back to LocalJobRunner Mode. Pig jobs were running fine with MRv1 but ever since the cluster was upgraded to use YARN, this problem is being noticed. I have ensured the classPATHs and environment variable are set right on this environment. We are using Cloudera Distribution (CDH 5.10) and pig version 0.12 (Apache Pig version 0.12.0-cdh5.10.2) Please assist with a solution - also do you think is there any incompatibility with PIG running with YARN in the recent or previous release of PIG or YARN? Thanks Asim Zafir. 2019-05-01 17:37:23,485 [main] DEBUG org.apache.pig.impl.util.JarManager - Adding entry pigContext to job jar 2019-05-01 17:37:23,721 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2836672534872456251.jar created 2019-05-01 17:37:23,721 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar 2019-05-01 17:37:23,742 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2019-05-01 17:37:23,745 [main] DEBUG org.apache.pig.data.SchemaTupleFrontend - Nothing registered to generate. *2019-05-01 17:37:23,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.* 2019-05-01 17:37:23,774 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address 2019-05-01 17:37:23,774 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2019-05-01 17:37:23,776 [JobControl] DEBUG org.apache.pig.backend.hadoop23.PigJobControl - Checking state of job job name: PigLatin:testrun.pig job id: job_pigexec_0 job state: WAITING job mapred id: null job message: just initialized job has no depending job:
