I tried to copy hadoop-mapreduce-client-2.4* from hadoop-2.4 to tez-0.4 and also copy to hdfs in /apps/tez/lib, and run Hive. But even if I set hive.execution.engine=tez or mr, jobs are running such as tez only :). (framework.name=yarn)
I know I have tried before hive-0.13 and tez-0.5 and someones said they are not compatible. Now I am running hive-0.13 + tez-0.4 + hadoop-2.4, but I am still not able to run correctly hive queries over tez or mapreduce. I already spent way too much time with this stuff and still is not running properly. Do you have any other suggestions, what to try ? On Thursday, June 19, 2014 4:30 PM, Hitesh Shah <[email protected]> wrote: Hi Robert, If you look at the tez jars, you will probably see a couple of mapreduce jars that Tez depends on under TEZ_BASE_DIR/lib/. Are these jars inconsistent with respect to the version of hadoop/mapreduce on the cluster? — Hitesh On Jun 19, 2014, at 2:59 PM, Grandl Robert <[email protected]> wrote: > Hmm, > > > Things are really weird. So If I don't add TEZ jars/conf in hadoop_classpath > simply running Mapreduce jobs works. If I add TEZ jars/conf to > hadoop_classpath and try to run mapreduce jobs, it fails with the exception: > 14/06/19 14:46:40 INFO mapreduce.JobSubmitter: Cleaning up the staging area > /tmp/hadoop-yarn/staging/hadoop/.staging/job_1403214320220_0001 > java.lang.NoSuchMethodError: > org.apache.hadoop.mapreduce.v2.util.MRApps.crossPlatformifyMREnv(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/yarn/api/ApplicationConstants$Environment;)Ljava/lang/String; > at >org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:390) > at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:284) > > > > If I add TEZ jars/conf to hadoop classpath BUT also change framework name > from yarn to yarn-tez and run tez application(like orderedwordcount) it > succeeds. Now I can also run hadoop mapreduce apps, but still as tez. I am > completely confused why I cannot run just mapreduce jobs(with framework name > = yarn) when adding TEZ in classpath. Is this a bug ? > > robert > > > > > On Thursday, June 19, 2014 2:34 PM, Jian He <[email protected]> wrote: > > > > The problem looks like it's referencing an old jar which doesn't have this > method. are you running on single node cluster? can you check you cluster > setting about jar dependency. If it's easy for you, just refresh the > environment and re-deploy the single node cluster. > > Jian > > > On Thu, Jun 19, 2014 at 9:15 AM, Grandl Robert <[email protected]> > wrote: > >> Any suggestion related to this ? >> >> Thanks, >> robert >> >> >> >> On Wednesday, June 18, 2014 11:56 PM, Grandl Robert >> <[email protected]> wrote: >> >> >> >> I am using 2.4 for client as well. Actually I took a tar gz from >> http://apache.petsads.us/hadoop/common/hadoop-2.4.0/, >> >> and I am trying to run. I tried even one node. I am lack of ideas why this >> happens. >> >> >> >> >> >> On Wednesday, June 18, 2014 11:52 PM, Jian He <[email protected]> wrote: >> >> >> >> This new method crossPlatformifyMREnv is newly added in 2.4.0 release, >> which version of MR client are you using? >> can you make sure you have the same version of client jars >> >> Jian >> >> >> >> On Wed, Jun 18, 2014 at 10:37 PM, Grandl Robert <[email protected]> >> wrote: >> >> Hi guys, >>> >>> I don't know what I did but my hadoop yarn went crazy. I am not able to >> submit any job, as it throws the following exception. >>> >>> 4/06/18 22:25:19 INFO mapreduce.JobSubmitter: number of splits:1 >>> 14/06/18 22:25:19 INFO mapreduce.JobSubmitter: Submitting tokens for job: >> job_1403155404621_0001 >>> 14/06/18 22:25:19 INFO mapreduce.JobSubmitter: Cleaning up the staging >> area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1403155404621_0001 >>> java.lang.NoSuchMethodError: >> org.apache.hadoop.mapreduce.v2.util.MRApps.crossPlatformifyMREnv(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/yarn/api/ApplicationConstants$Environment;)Ljava/lang/String; >>> at >> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:390) >>> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:284) >>> at >> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430) >>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) >>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) >>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) >>> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286) >>> at >> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306) >>> at >> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>> at >> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:601) >>> at >> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) >>> >>> >>> I configured all class path and variables as such: >>> export HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.4.0 >>> export HADOOP_HOME=$HADOOP_COMMON_HOME >>> export HADOOP_HDFS_HOME=$HADOOP_COMMON_HOME >>> export HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME >>> export HADOOP_YARN_HOME=$HADOOP_COMMON_HOME >>> export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_MAPRED_HOME/lib/native/ >>> export HADOOP_CONF_DIR=/home/hadoop/rgrandl/conf/ >>> export YARN_CONF_DIR=/home/hadoop/rgrandl/conf/ >>> export HADOOP_BIN_PATH=$HADOOP_MAPRED_HOME/bin/ >>> export HADOOP_SBIN=$HADOOP_MAPRED_HOME/sbin/ >>> export HADOOP_LOGS=$HADOOP_HOME/logs >>> export HADOOP_LOG_DIR=$HADOOP_HOME/logs >>> export YARN_LOG_DIR=$HADOOP_HOME/logs >>> >>> export JAVA_HOME=/home/hadoop/rgrandl/java/ >>> export HADOOP_USER_CLASSPATH_FIRST=1 >>> export YARN_HOME=/home/hadoop/hadoop-2.4.0 >>> export TEZ_CONF_DIR=/home/hadoop/rgrandl/conf >>> export TEZ_JARS=/home/hadoop/rgrandl/tez/tez-0.4.0-incubating >>> >>> export HADOOP_PREFIX=$HADOOP_COMMON_HOME >>> >>> export >> HADOOP_CLASSPATH=$HADOOP_HOME:/home/hadoop/rgrandl/tez/tez-0.4.0-incubating/*:/home/hadoop/rgrandl/tez/tez-0.4.0-incubating/lib/*:/home/hadoop/rgrandl/hive:/home/hadoop/rgrandl/conf >>> >>> export >> PATH=$PATH:$HADOOP_BIN_PATH:$HADOOP_SBIN:$YARN_CONF_DIR:$HADOOP_YARN_HOME:$HADOOP_MAPRED_HOME:$HADOOP_HDFS_HOME:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME:$JAVA_HOME/bin/:/home/hadoop/rgrandl/hive/bin >>> >>> >>> Everything seems to be correct, but I cannot understand this error. Is >> something I never encountered before. >>> >>> >>> Do you have any hints on it ? >>> >>> Thanks, >>> robert >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. > >> > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
