Another issue - when running tests I'm seeing lots of errors related to
incompatibilities between hadoop versions.
Stepping back, can someone help me understand what "HCatalog hadoop23
support" means? The reason for asking is because I'm putting together a
change to add hadoop23 support to the build using maven profiles, and I'm
running into a number of issues and am confused about what the expected
suport level is (like do we expect tests to run? which submodules build
with 23, ...).
Testcase: testPartedRead took 1.121 sec
Caused an ERROR
org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:258)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1208)
at org.apache.pig.PigServer.execute(PigServer.java:1200)
at org.apache.pig.PigServer.access$100(PigServer.java:128)
at org.apache.pig.PigServer$Graph.execute(PigServer.java:1527)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1574)
at org.apache.pig.PigServer.registerQuery(PigServer.java:533)
at org.apache.pig.PigServer.registerQuery(PigServer.java:557)
at
org.apache.hcatalog.mapreduce.HCatBaseTest.logAndRegister(HCatBaseTest.java:83)
at
org.apache.hcatalog.mapreduce.TestHCatHiveCompatibility.testPartedRead(TestHCatHiveCompatibility.java:101)
Thanks!
Travis
On Fri, Nov 16, 2012 at 9:55 AM, Travis Crawford <[email protected]>
wrote:
> Another hadoop23 question - would you expect hbase-storage-handler
> tests to compile against hadoop23? I'm seeing issues with
> SkeletonHBaseTest.
>
> FileUtil.fullyDelete changed behavior between hadoop versions:
>
> hadoop20: throws an IOException
> hadoop23: does not throw any exceptions
>
> Trying to understand what the expected hadoop version support is for
> hbase-storage-handler.
>
> Thanks!
> Travis
>
>
>
> On Thu, Nov 15, 2012 at 8:39 PM, Travis Crawford
> <[email protected]> wrote:
>> Awesome - thx for the info!
>>
>> --travis
>>
>>
>> On Thu, Nov 15, 2012 at 7:46 PM, Francis Liu <[email protected]> wrote:
>>> There's no support yet. Yep let's keep it building against 20 for now.
>>>
>>>
>>> On Nov 15, 2012, at 2:05 PM, Travis Crawford <[email protected]>
wrote:
>>>
>>>> Hey hcat gurus -
>>>>
>>>> Does webhcat work with hadoop 23? I'm not super familiar with yarn
yet, and
>>>> am not seeing the JobTracker class at all, so it looks like there's no
>>>> backwards-compatibility mode.
>>>>
>>>> Does this mean that webhcat server should always build with 20 for the
time
>>>> being?
>>>>
>>>> compile:
>>>> [echo] webhcat
>>>> [mkdir] Created dir:
>>>> /Users/travis/src/hcatalog/webhcat/svr/build/classes
>>>> [javac] Compiling 56 source files to
>>>> /Users/travis/src/hcatalog/webhcat/svr/build/classes
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:32:
>>>> cannot find symbol
>>>> [javac] symbol : class JobSubmissionProtocol
>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker
>>>> [javac] private JobSubmissionProtocol cnx;
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/DeleteDelegator.java:22:
>>>> cannot find symbol
>>>> [javac] symbol : class JobTracker
>>>> [javac] location: package org.apache.hadoop.mapred
>>>> [javac] import org.apache.hadoop.mapred.JobTracker;
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/ListDelegator.java:25:
>>>> cannot find symbol
>>>> [javac] symbol : class JobTracker
>>>> [javac] location: package org.apache.hadoop.mapred
>>>> [javac] import org.apache.hadoop.mapred.JobTracker;
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/StatusDelegator.java:27:
>>>> cannot find symbol
>>>> [javac] symbol : class JobTracker
>>>> [javac] location: package org.apache.hadoop.mapred
>>>> [javac] import org.apache.hadoop.mapred.JobTracker;
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:41:
>>>> cannot find symbol
>>>> [javac] symbol : class JobSubmissionProtocol
>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker
>>>> [javac] cnx = (JobSubmissionProtocol)
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:42:
>>>> cannot find symbol
>>>> [javac] symbol : class JobSubmissionProtocol
>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker
>>>> [javac] RPC.getProxy(JobSubmissionProtocol.class,
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:43:
>>>> cannot find symbol
>>>> [javac] symbol : variable JobSubmissionProtocol
>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker
>>>> [javac] JobSubmissionProtocol.versionID,
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hadoop/mapred/TempletonJobTracker.java:48:
>>>> cannot find symbol
>>>> [javac] symbol : class JobSubmissionProtocol
>>>> [javac] location: class org.apache.hadoop.mapred.TempletonJobTracker
>>>> [javac] JobSubmissionProtocol.class));
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/DeleteDelegator.java:43:
>>>> cannot find symbol
>>>> [javac] symbol : variable JobTracker
>>>> [javac] location: class
org.apache.hcatalog.templeton.DeleteDelegator
>>>> [javac]
>>>> JobTracker.getAddress(appConf),
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/ListDelegator.java:44:
>>>> cannot find symbol
>>>> [javac] symbol : variable JobTracker
>>>> [javac] location: class org.apache.hcatalog.templeton.ListDelegator
>>>> [javac] JobTracker.getAddress(appConf),
>>>> [javac] ^
>>>> [javac]
>>>>
/Users/travis/src/hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/StatusDelegator.java:49:
>>>> cannot find symbol
>>>> [javac] symbol : variable JobTracker
>>>> [javac] location: class
org.apache.hcatalog.templeton.StatusDelegator
>>>> [javac] JobTracker.getAddress(appConf),
>>>> [javac] ^
>>>> [javac] Note: Some input files use or override a deprecated API.
>>>> [javac] Note: Recompile with -Xlint:deprecation for details.
>>>> [javac] Note: Some input files use unchecked or unsafe operations.
>>>> [javac] Note: Recompile with -Xlint:unchecked for details.
>>>> [javac] 11 errors
>>>>
>>>> Thanks!
>>>> Travis
>>>
