No worries, thx for cluing me into what's expected to work. I'm testing a branch now and should have a patch adding 23 support back shortly.
--travis On Fri, Nov 16, 2012 at 1:07 PM, Francis Liu <[email protected]> wrote: > > Right now hadoop 23 only works for compiling main code. We did not attempt to > get tests to compile/pass since it is blocked by two things: 1. antlr > conflict between Pig and Hive and 2. Pig publishing 23 jars. These are > captured in HCAT jiras. Apologies for not mentioning it previously. > > -Francis > > On Nov 16, 2012, at 10:09 AM, Travis Crawford <[email protected]> > wrote: > > > Another issue - when running tests I'm seeing lots of errors related to > > incompatibilities between hadoop versions. > > > > Stepping back, can someone help me understand what "HCatalog hadoop23 > > support" means? The reason for asking is because I'm putting together a > > change to add hadoop23 support to the build using maven profiles, and I'm > > running into a number of issues and am confused about what the expected > > suport level is (like do we expect tests to run? which submodules build > > with 23, ...). > > > > Testcase: testPartedRead took 1.121 sec > > Caused an ERROR > > org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String; > > java.lang.NoSuchMethodError: > > org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String; > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:258) > > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147) > > at > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378) > > at > > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1208) > > at org.apache.pig.PigServer.execute(PigServer.java:1200) > > at org.apache.pig.PigServer.access$100(PigServer.java:128) > > at org.apache.pig.PigServer$Graph.execute(PigServer.java:1527) > > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1574) > > at org.apache.pig.PigServer.registerQuery(PigServer.java:533) > > at org.apache.pig.PigServer.registerQuery(PigServer.java:557) > > at > > org.apache.hcatalog.mapreduce.HCatBaseTest.logAndRegister(HCatBaseTest.java:83) > > at > > org.apache.hcatalog.mapreduce.TestHCatHiveCompatibility.testPartedRead(TestHCatHiveCompatibility.java:101) > > > > Thanks! > > Travis > > > > > > On Fri, Nov 16, 2012 at 9:55 AM, Travis Crawford <[email protected]> > > wrote: > >> Another hadoop23 question - would you expect hbase-storage-handler > >> tests to compile against hadoop23? I'm seeing issues with > >> SkeletonHBaseTest. > >> > >> FileUtil.fullyDelete changed behavior between hadoop versions: > >> > >> hadoop20: throws an IOException > >> hadoop23: does not throw any exceptions > >> > >> Trying to understand what the expected hadoop version support is for > >> hbase-storage-handler. > >> > >> Thanks! > >> Travis > >> > >> > >> > >> On Thu, Nov 15, 2012 at 8:39 PM, Travis Crawford > >> <[email protected]> wrote: > >>> Awesome - thx for the info! > >>> > >>> --travis > >>> > >>> > >>> On Thu, Nov 15, 2012 at 7:46 PM, Francis Liu <[email protected]> wrote: > >>>> There's no support yet. Yep let's keep it building against 20 for now. > >>>> > >>>> > >>>> On Nov 15, 2012, at 2:05 PM, Travis Crawford <[email protected]> > > wrote: > >>>> > >>>>> Hey hcat gurus - > >>>>> > >>>>> Does webhcat work with hadoop 23? I'm not super familiar with yarn > > yet, and > >>>>> am not seeing the JobTracker class at all, so it looks like there's no > >>>>> backwards-compatibility mode. > >>>>> > >>>>> Does this mean that webhcat server should always build with 20 for the > > time > >>>>> being? > >>>>> > >>>>> compile: > >>>>> [echo] webhcat > >>>>> [mkdir] Created dir: > >>>>> /Users/travis/src/hcatalog/webhcat/svr/build/classes > >>>>> [javac] Compiling 56 source files to > >>>>> /Users/travis/src/hcatalog/webhcat/svr/build/classes > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:32: > >>>>> cannot find symbol > >>>>> [javac] symbol : class JobSubmissionProtocol > >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker > >>>>> [javac] private JobSubmissionProtocol cnx; > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/DeleteDelegator.java:22: > >>>>> cannot find symbol > >>>>> [javac] symbol : class JobTracker > >>>>> [javac] location: package org.apache.hadoop.mapred > >>>>> [javac] import org.apache.hadoop.mapred.JobTracker; > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/ListDelegator.java:25: > >>>>> cannot find symbol > >>>>> [javac] symbol : class JobTracker > >>>>> [javac] location: package org.apache.hadoop.mapred > >>>>> [javac] import org.apache.hadoop.mapred.JobTracker; > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/StatusDelegator.java:27: > >>>>> cannot find symbol > >>>>> [javac] symbol : class JobTracker > >>>>> [javac] location: package org.apache.hadoop.mapred > >>>>> [javac] import org.apache.hadoop.mapred.JobTracker; > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:41: > >>>>> cannot find symbol > >>>>> [javac] symbol : class JobSubmissionProtocol > >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker > >>>>> [javac] cnx = (JobSubmissionProtocol) > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:42: > >>>>> cannot find symbol > >>>>> [javac] symbol : class JobSubmissionProtocol > >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker > >>>>> [javac] RPC.getProxy(JobSubmissionProtocol.class, > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:43: > >>>>> cannot find symbol > >>>>> [javac] symbol : variable JobSubmissionProtocol > >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker > >>>>> [javac] JobSubmissionProtocol.versionID, > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:48: > >>>>> cannot find symbol > >>>>> [javac] symbol : class JobSubmissionProtocol > >>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker > >>>>> [javac] JobSubmissionProtocol.class)); > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/DeleteDelegator.java:43: > >>>>> cannot find symbol > >>>>> [javac] symbol : variable JobTracker > >>>>> [javac] location: class > > org.apache.hcatalog.templeton.DeleteDelegator > >>>>> [javac] > >>>>> JobTracker.getAddress(appConf), > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/ListDelegator.java:44: > >>>>> cannot find symbol > >>>>> [javac] symbol : variable JobTracker > >>>>> [javac] location: class org.apache.hcatalog.templeton.ListDelegator > >>>>> [javac] JobTracker.getAddress(appConf), > >>>>> [javac] ^ > >>>>> [javac] > >>>>> > > /Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/StatusDelegator.java:49: > >>>>> cannot find symbol > >>>>> [javac] symbol : variable JobTracker > >>>>> [javac] location: class > > org.apache.hcatalog.templeton.StatusDelegator > >>>>> [javac] JobTracker.getAddress(appConf), > >>>>> [javac] ^ > >>>>> [javac] Note: Some input files use or override a deprecated API. > >>>>> [javac] Note: Recompile with -Xlint:deprecation for details. > >>>>> [javac] Note: Some input files use unchecked or unsafe operations. > >>>>> [javac] Note: Recompile with -Xlint:unchecked for details. > >>>>> [javac] 11 errors > >>>>> > >>>>> Thanks! > >>>>> Travis > >>>> >
