Right now hadoop 23 only works for compiling main code. We did not attempt to get tests to compile/pass since it is blocked by two things: 1. antlr conflict between Pig and Hive and 2. Pig publishing 23 jars. These are captured in HCAT jiras. Apologies for not mentioning it previously.
-Francis On Nov 16, 2012, at 10:09 AM, Travis Crawford <[email protected]> wrote: > Another issue - when running tests I'm seeing lots of errors related to > incompatibilities between hadoop versions. > > Stepping back, can someone help me understand what "HCatalog hadoop23 > support" means? The reason for asking is because I'm putting together a > change to add hadoop23 support to the build using maven profiles, and I'm > running into a number of issues and am confused about what the expected > suport level is (like do we expect tests to run? which submodules build > with 23, ...). > > Testcase: testPartedRead took 1.121 sec > Caused an ERROR > org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String; > java.lang.NoSuchMethodError: > org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String; > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:258) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147) > at > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378) > at > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1208) > at org.apache.pig.PigServer.execute(PigServer.java:1200) > at org.apache.pig.PigServer.access$100(PigServer.java:128) > at org.apache.pig.PigServer$Graph.execute(PigServer.java:1527) > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1574) > at org.apache.pig.PigServer.registerQuery(PigServer.java:533) > at org.apache.pig.PigServer.registerQuery(PigServer.java:557) > at > org.apache.hcatalog.mapreduce.HCatBaseTest.logAndRegister(HCatBaseTest.java:83) > at > org.apache.hcatalog.mapreduce.TestHCatHiveCompatibility.testPartedRead(TestHCatHiveCompatibility.java:101) > > Thanks! > Travis > > > On Fri, Nov 16, 2012 at 9:55 AM, Travis Crawford <[email protected]> > wrote: >> Another hadoop23 question - would you expect hbase-storage-handler >> tests to compile against hadoop23? I'm seeing issues with >> SkeletonHBaseTest. >> >> FileUtil.fullyDelete changed behavior between hadoop versions: >> >> hadoop20: throws an IOException >> hadoop23: does not throw any exceptions >> >> Trying to understand what the expected hadoop version support is for >> hbase-storage-handler. >> >> Thanks! >> Travis >> >> >> >> On Thu, Nov 15, 2012 at 8:39 PM, Travis Crawford >> <[email protected]> wrote: >>> Awesome - thx for the info! >>> >>> --travis >>> >>> >>> On Thu, Nov 15, 2012 at 7:46 PM, Francis Liu <[email protected]> wrote: >>>> There's no support yet. Yep let's keep it building against 20 for now. >>>> >>>> >>>> On Nov 15, 2012, at 2:05 PM, Travis Crawford <[email protected]> > wrote: >>>> >>>>> Hey hcat gurus - >>>>> >>>>> Does webhcat work with hadoop 23? I'm not super familiar with yarn > yet, and >>>>> am not seeing the JobTracker class at all, so it looks like there's no >>>>> backwards-compatibility mode. >>>>> >>>>> Does this mean that webhcat server should always build with 20 for the > time >>>>> being? >>>>> >>>>> compile: >>>>> [echo] webhcat >>>>> [mkdir] Created dir: >>>>> /Users/travis/src/hcatalog/webhcat/svr/build/classes >>>>> [javac] Compiling 56 source files to >>>>> /Users/travis/src/hcatalog/webhcat/svr/build/classes >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:32: >>>>> cannot find symbol >>>>> [javac] symbol : class JobSubmissionProtocol >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker >>>>> [javac] private JobSubmissionProtocol cnx; >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/DeleteDelegator.java:22: >>>>> cannot find symbol >>>>> [javac] symbol : class JobTracker >>>>> [javac] location: package org.apache.hadoop.mapred >>>>> [javac] import org.apache.hadoop.mapred.JobTracker; >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/ListDelegator.java:25: >>>>> cannot find symbol >>>>> [javac] symbol : class JobTracker >>>>> [javac] location: package org.apache.hadoop.mapred >>>>> [javac] import org.apache.hadoop.mapred.JobTracker; >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/StatusDelegator.java:27: >>>>> cannot find symbol >>>>> [javac] symbol : class JobTracker >>>>> [javac] location: package org.apache.hadoop.mapred >>>>> [javac] import org.apache.hadoop.mapred.JobTracker; >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:41: >>>>> cannot find symbol >>>>> [javac] symbol : class JobSubmissionProtocol >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker >>>>> [javac] cnx = (JobSubmissionProtocol) >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:42: >>>>> cannot find symbol >>>>> [javac] symbol : class JobSubmissionProtocol >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker >>>>> [javac] RPC.getProxy(JobSubmissionProtocol.class, >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:43: >>>>> cannot find symbol >>>>> [javac] symbol : variable JobSubmissionProtocol >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker >>>>> [javac] JobSubmissionProtocol.versionID, >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:48: >>>>> cannot find symbol >>>>> [javac] symbol : class JobSubmissionProtocol >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker >>>>> [javac] JobSubmissionProtocol.class)); >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/DeleteDelegator.java:43: >>>>> cannot find symbol >>>>> [javac] symbol : variable JobTracker >>>>> [javac] location: class > org.apache.hcatalog.templeton.DeleteDelegator >>>>> [javac] >>>>> JobTracker.getAddress(appConf), >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/ListDelegator.java:44: >>>>> cannot find symbol >>>>> [javac] symbol : variable JobTracker >>>>> [javac] location: class org.apache.hcatalog.templeton.ListDelegator >>>>> [javac] JobTracker.getAddress(appConf), >>>>> [javac] ^ >>>>> [javac] >>>>> > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/StatusDelegator.java:49: >>>>> cannot find symbol >>>>> [javac] symbol : variable JobTracker >>>>> [javac] location: class > org.apache.hcatalog.templeton.StatusDelegator >>>>> [javac] JobTracker.getAddress(appConf), >>>>> [javac] ^ >>>>> [javac] Note: Some input files use or override a deprecated API. >>>>> [javac] Note: Recompile with -Xlint:deprecation for details. >>>>> [javac] Note: Some input files use unchecked or unsafe operations. >>>>> [javac] Note: Recompile with -Xlint:unchecked for details. >>>>> [javac] 11 errors >>>>> >>>>> Thanks! >>>>> Travis >>>>
