Hi Folks,
I am fairly new to Hadoop MR2 environment. I have Hadoop0.23.1 (single
node) on Linux box and can successfully execute YARN test job, but when I
execute PIG (Ver. 12) to access and process files at HDFS, generate error
msg as follows. I recompiled PIG with Hadoop23, but still no luck.
ant clean jar-withouthadoop -Dhadoopversion=23
Can someone please help and advise what I am missing here !!
Note: PIG in local mode works without any issue.
================
*grunt> A = load 'NYSE_dividends' as (exch, symb, dt, div);*
2013-11-26 13:20:23,151 [main] WARN org.apache.hadoop.conf.
Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,151 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:23,323 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,323 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
*grunt> dump A ;*
2013-11-26 13:20:31,805 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:31,806 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:31,841 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2013-11-26 13:20:31,913 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune,
DuplicateForEachColumnRewrite, GroupByConstParallelSetter,
ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,
MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter],
RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2013-11-26 13:20:32,159 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,266 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2013-11-26 13:20:32,304 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,304 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,318 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2013-11-26 13:20:32,319 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2013-11-26 13:20:32,322 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,322 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,498 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2013-11-26 13:20:32,504 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-11-26 13:20:32,504 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.output.compress is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress
2013-11-26 13:20:32,506 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job1088179330840878220.jar
2013-11-26 13:20:35,473 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job1088179330840878220.jar created
2013-11-26 13:20:35,473 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.jar is deprecated. Instead, use mapreduce.job.jar
2013-11-26 13:20:35,488 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2013-11-26 13:20:35,492 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Map only job, skipping reducer estimation
2013-11-26 13:20:35,513 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2013-11-26 13:20:35,514 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.http.address is deprecated. Instead, use
mapreduce.jobtracker.http.address
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library not
loaded
2013-11-26 13:20:35,973 [JobControl] INFO
org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging area
null/hadoop/.staging/job_1385489868206_0001
*2013-11-26 13:20:35,975 [JobControl] ERROR
org.apache.pig.backend.hadoop23.PigJobControl - Error while trying to run
jobs.java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException*
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
at java.lang.Thread.run(Thread.java:619)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
*Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)*
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
... 3 more
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileSystem.getDefaultBlockSize(Lorg/apache/hadoop/fs/Path;)J
at
org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getDefaultBlockSize(HadoopShims.java:108)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:277)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:451)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:468)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
... 8 more
2013-11-26 13:20:35,976 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_1385489868206_0001
2013-11-26 13:20:35,976 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases A
2013-11-26 13:20:35,976 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: A[1,4] C: R:
2013-11-26 13:20:35,979 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2013-11-26 13:20:35,983 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to
stop immediately on failure.
2013-11-26 13:20:35,983 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_1385489868206_0001 has failed! Stop running all dependent jobs
2013-11-26 13:20:35,983 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2013-11-26 13:20:36,104 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Exception thrown by remote
end.
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: Trying
to get information for an absent application application_1385489868206_0001
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:150)
at $Proxy7.getApplicationReport(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClientImpl.java:116)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:343)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:142)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:288)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:365)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:483)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:186)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:627)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:625)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:625)
at
org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:679)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:673)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:150)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:428)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.storeEx(PigServer.java:978)
at org.apache.pig.PigServer.store(PigServer.java:942)
at org.apache.pig.PigServer.openIterator(PigServer.java:855)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:541)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
2013-11-26 13:20:36,105 [main] ERROR
org.apache.hadoop.security.UserGroupInformation -
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Trying to get information for an absent application
application_1385489868206_0001
2013-11-26 13:20:36,105 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR: Trying to get
information for an absent application application_1385489868206_0001
2013-11-26 13:20:36,105 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2013-11-26 13:20:36,107 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt
Features
0.23.1 0.12.1-SNAPSHOT hadoop 2013-11-26 13:20:32 2013-11-26
13:20:36 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_1385489868206_0001 A MAP_ONLY Message: Unexpected System
Error Occured: java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
at java.lang.Thread.run(Thread.java:619)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
... 3 more
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileSystem.getDefaultBlockSize(Lorg/apache/hadoop/fs/Path;)J
at
org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getDefaultBlockSize(HadoopShims.java:108)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:277)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:451)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:468)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
... 8 more
hdfs://sasserver05:54310/tmp/temp601167191/tmp-1605231029,
Input(s):
Failed to read data from
"hdfs://sasserver05:54310/user/hadoop/NYSE_dividends"
Output(s):
Failed to produce result in
"hdfs://sasserver05:54310/tmp/temp601167191/tmp-1605231029"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1385489868206_0001
2013-11-26 13:20:36,107 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2013-11-26 13:20:36,112 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1066: Unable to open iterator for alias A. Backend error : Trying to
get information for an absent application application_1385489868206_0001
Details at logfile: /home/hadoop/pig_1385489876211.log
grunt>
=============
On Tue, Nov 26, 2013 at 3:32 PM, Claudio Romo Otto <
claudio.r...@namictech.com> wrote:
> Hi Juan,
>
> In a nutshell, you must pay attention to memory settings inside
> mapred-site.xml, yarn-site.xml, hadoop-env.sh and yarn-env.sh, so you have
> to design a memory distribution strategy according to your performance
> requirements. In this way you will have, among other things, enough memory
> for the Scheduler.
>
> Remember to reserve at least 600 - 800 mb for the operative system to
> avoid OOM errors.
>
> Best regards
> El 26/11/13 16:07, Juan Martin Pampliega escribió:
>
>> Hi Claudio,
>>
>> It would be nice to know which were the settings that you had to tune to
>> get this. I am having a similar issue with some jobs that I am running.
>> Thanks,
>> Juan.
>>
>>
>> On Wed, Oct 30, 2013 at 7:40 PM, Claudio Romo Otto <
>> claudio.r...@namictech.com> wrote:
>>
>> Jarcec, finally I got solved this problem by learning more on hadoop 2
>>> (lot of reading), and then tuning some settings to let the work move from
>>> the SCHEDULED state. With this said, the last problem was only concerning
>>> on hadoop.
>>>
>>> Thanks for your support!
>>>
>>> El 30/10/13 18:03, Jarek Jarcec Cecho escribió:
>>>
>>> Hi Claudio,
>>>
>>>> it's hard to guess from the limited information. I would suggest to take
>>>> a look into logs to see what is happening.
>>>>
>>>> One guess though - You've mentioned that the task was "running" for 30
>>>> minutes, but it still seems to be in SCHEDULED time - are your node
>>>> managers correctly running?
>>>>
>>>> Jarcec
>>>>
>>>> On Fri, Oct 25, 2013 at 04:10:12PM -0300, Claudio Romo Otto wrote:
>>>>
>>>> You got it!
>>>>>
>>>>> The solution was to compile with -Dhadoopversion=23 option. After
>>>>> your message I tried another test removing Cassandra from the chain
>>>>> and Pig sent successfully the job to hadoop.
>>>>>
>>>>> BUT! the problem changed, now the Map task remains forever stuck on
>>>>> Hadoop (30 minutes waiting, no other jobs running):
>>>>>
>>>>> Task
>>>>>
>>>>> Progress
>>>>>
>>>>> State
>>>>>
>>>>> Start Time
>>>>>
>>>>> Finish Time
>>>>>
>>>>> Elapsed Time
>>>>> task_1382631533263_0012_m_000000 <http://topgps-test-3.
>>>>> dnsalias.com:8088/proxy/application_1382631533263_
>>>>> 0012/mapreduce/task/task_1382631533263_0012_m_000000>
>>>>>
>>>>> SCHEDULED Fri, 25 Oct 2013 18:18:32 GMT N/A 0sec
>>>>>
>>>>>
>>>>>
>>>>> Attempt
>>>>>
>>>>> Progress
>>>>>
>>>>> State
>>>>>
>>>>> Node
>>>>>
>>>>> Logs
>>>>>
>>>>> Started
>>>>>
>>>>> Finished
>>>>>
>>>>> Elapsed
>>>>>
>>>>> Note
>>>>> attempt_1382631533263_0012_m_000000_0 0,00 STARTING N/A
>>>>> N/A N/A
>>>>> N/A 0sec
>>>>>
>>>>>
>>>>> Don't know if this is a Hadoop problem or Pig, what do you think?
>>>>>
>>>>>
>>>>> El 25/10/13 13:11, Jarek Jarcec Cecho escribió:
>>>>>
>>>>> It seems that Pig was correctly compiled against Hadoop 23, but the
>>>>>> Cassandra piece was not, check out the where the exception is coming
>>>>>> from:
>>>>>>
>>>>>> Caused by: java.lang.IncompatibleClassChangeError: Found interface
>>>>>>
>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>>> at org.apache.cassandra.hadoop.AbstractColumnFamilyInputForma
>>>>>>> t.getSplits(AbstractColumnFamilyInputFormat.java:113)
>>>>>>>
>>>>>>> So, I would say that you also need to get Hadoop 2 compatible
>>>>>> Cassandra
>>>>>> connector first.
>>>>>>
>>>>>> Jarcec
>>>>>>
>>>>>> On Thu, Oct 24, 2013 at 10:34:49PM -0300, Claudio Romo Otto wrote:
>>>>>>
>>>>>> After change from hadoop20 to hadoop23 the warning dissapeared but I
>>>>>>> got the same exception (Found interface
>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected)
>>>>>>>
>>>>>>> I have tried over a fresh install: hadoop 2.2.0 and pig 0.12.1
>>>>>>> compiled by me, no other product nor configuration, just two
>>>>>>> servers, one master with ResourceManager and NameNode, one slave
>>>>>>> with DataNode and NodeManager.
>>>>>>>
>>>>>>> I can't understand why over this fresh cluster Pig 0.12 fails. Here
>>>>>>> is the new trace:
>>>>>>>
>>>>>>> 2013-10-24 16:10:52,351 [JobControl] ERROR
>>>>>>> org.apache.pig.backend.hadoop23.PigJobControl - Error while trying
>>>>>>> to run jobs.
>>>>>>> java.lang.RuntimeException: java.lang.reflect.
>>>>>>> InvocationTargetException
>>>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
>>>>>>> PigJobControl.java:130)
>>>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.run(
>>>>>>> PigJobControl.java:191)
>>>>>>> at java.lang.Thread.run(Thread.java:724)
>>>>>>> at org.apache.pig.backend.hadoop.executionengine.
>>>>>>> mapReduceLayer.
>>>>>>> MapReduceLauncher$1.run(MapReduceLauncher.java:257)
>>>>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>>> NativeMethodAccessorImpl.java:57)
>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>>> DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
>>>>>>> PigJobControl.java:128)
>>>>>>> ... 3 more
>>>>>>> Caused by: java.lang.IncompatibleClassChangeError: Found interface
>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>>> at org.apache.cassandra.hadoop.AbstractColumnFamilyInputForma
>>>>>>> t.getSplits(AbstractColumnFamilyInputFormat.java:113)
>>>>>>> at org.apache.pig.backend.hadoop.executionengine.
>>>>>>> mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
>>>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(
>>>>>>> JobSubmitter.java:491)
>>>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(
>>>>>>> JobSubmitter.java:508)
>>>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
>>>>>>> JobSubmitter.java:392)
>>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(
>>>>>>> UserGroupInformation.java:1491)
>>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>>>>> at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.
>>>>>>> submit(ControlledJob.java:335)
>>>>>>> ... 8 more
>>>>>>>
>>>>>>>
>>>>>>> El 24/10/13 21:33, Prashant Kommireddi escribió:
>>>>>>>
>>>>>>> Yes it does. You need to recompile Pig for hadoop 2
>>>>>>>>
>>>>>>>> ant clean jar-withouthadoop -Dhadoopversion=23
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 24, 2013 at 5:37 AM, Claudio Romo Otto <
>>>>>>>> claudio.r...@namictech.com> wrote:
>>>>>>>>
>>>>>>>> Does Pig support Hadoop 2.2? When I try Pig 0.12 and Hadoop 2.2 I
>>>>>>>>
>>>>>>>>> get an
>>>>>>>>> error even with simple operations like
>>>>>>>>>
>>>>>>>>> data = LOAD 'cql://keyspace1/testcf?' USING CqlStorage();
>>>>>>>>> dump data;
>>>>>>>>>
>>>>>>>>> I only got a warning first and then and exception:
>>>>>>>>>
>>>>>>>>> 2013-10-24 09:35:19,300 [main] WARN org.apache.pig.backend.**
>>>>>>>>> hadoop20.PigJobControl
>>>>>>>>> - falling back to default JobControl (not using hadoop 0.20 ?)
>>>>>>>>> java.lang.**NoSuchFieldException: runnerState
>>>>>>>>> at java.lang.Class.**getDeclaredField(Class.java:**1938)
>>>>>>>>> at org.apache.pig.backend.**hadoop20.PigJobControl.<**
>>>>>>>>> clinit>(PigJobControl.java:51)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.shims.**
>>>>>>>>> HadoopShims.newJobControl(**HadoopShims.java:97)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**
>>>>>>>>> JobControlCompiler.compile(**JobControlCompiler.java:285)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**
>>>>>>>>> MapReduceLauncher.launchPig(**MapReduceLauncher.java:177)
>>>>>>>>> at org.apache.pig.PigServer.**launchPlan(PigServer.java:**
>>>>>>>>> 1264)
>>>>>>>>> at org.apache.pig.PigServer.**executeCompiledLogicalPlan(**
>>>>>>>>> PigServer.java:1249)
>>>>>>>>> at org.apache.pig.PigServer.**storeEx(PigServer.java:931)
>>>>>>>>> at org.apache.pig.PigServer.**store(PigServer.java:898)
>>>>>>>>> at org.apache.pig.PigServer.**openIterator(PigServer.java:**
>>>>>>>>> 811)
>>>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.processDump(**
>>>>>>>>> GruntParser.java:696)
>>>>>>>>> at org.apache.pig.tools.**pigscript.parser.**
>>>>>>>>> PigScriptParser.parse(**
>>>>>>>>> PigScriptParser.java:320)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>>> GruntParser.java:194)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>>> GruntParser.java:170)
>>>>>>>>> at org.apache.pig.tools.grunt.**Grunt.run(Grunt.java:69)
>>>>>>>>> at org.apache.pig.Main.run(Main.**java:538)
>>>>>>>>> at org.apache.pig.Main.main(Main.**java:157)
>>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native
>>>>>>>>> Method)
>>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>>>>>>>>> NativeMethodAccessorImpl.java:**57)
>>>>>>>>> at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>>>>>>>>> DelegatingMethodAccessorImpl.**java:43)
>>>>>>>>> at java.lang.reflect.Method.**invoke(Method.java:606)
>>>>>>>>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:212)
>>>>>>>>>
>>>>>>>>> ------------------------------**--------
>>>>>>>>>
>>>>>>>>> Backend error message during job submission
>>>>>>>>> ------------------------------**-------------
>>>>>>>>> Unexpected System Error Occured: java.lang.**
>>>>>>>>> IncompatibleClassChangeError:
>>>>>>>>> Found interface org.apache.hadoop.mapreduce.**JobContext, but
>>>>>>>>> class
>>>>>>>>> was
>>>>>>>>> expected
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**PigOutputFormat.**setupUdfEnvAndStores(**
>>>>>>>>> PigOutputFormat.java:225)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**PigOutputFormat.**checkOutputSpecs(**
>>>>>>>>> PigOutputFormat.java:186)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> JobSubmitter.checkSpecs(**
>>>>>>>>> JobSubmitter.java:456)
>>>>>>>>> at org.apache.hadoop.mapreduce.**JobSubmitter.**
>>>>>>>>> submitJobInternal(
>>>>>>>>> **JobSubmitter.java:342)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> Job$10.run(Job.java:1268)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> Job$10.run(Job.java:1265)
>>>>>>>>> at java.security.**AccessController.doPrivileged(
>>>>>>>>> **Native
>>>>>>>>> Method)
>>>>>>>>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>>>>>>>>> at org.apache.hadoop.security.**
>>>>>>>>> UserGroupInformation.doAs(**
>>>>>>>>> UserGroupInformation.java:**1491)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> Job.submit(Job.java:1265)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> lib.jobcontrol.ControlledJob.**
>>>>>>>>> submit(ControlledJob.java:335)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> lib.jobcontrol.JobControl.run(**
>>>>>>>>> JobControl.java:240)
>>>>>>>>> at org.apache.pig.backend.**
>>>>>>>>> hadoop20.PigJobControl.run(**
>>>>>>>>> PigJobControl.java:121)
>>>>>>>>> at java.lang.Thread.run(Thread.**java:724)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**MapReduceLauncher$1.run(**
>>>>>>>>> MapReduceLauncher.java:257)
>>>>>>>>>
>>>>>>>>> Pig Stack Trace
>>>>>>>>> ---------------
>>>>>>>>> ERROR 1066: Unable to open iterator for alias data
>>>>>>>>>
>>>>>>>>> org.apache.pig.impl.**logicalLayer.**FrontendException: ERROR
>>>>>>>>> 1066:
>>>>>>>>> Unable to open iterator for alias data
>>>>>>>>> at org.apache.pig.PigServer.**
>>>>>>>>> openIterator(PigServer.java:**836)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.processDump(**
>>>>>>>>> GruntParser.java:696)
>>>>>>>>> at org.apache.pig.tools.**pigscript.parser.**
>>>>>>>>> PigScriptParser.parse(**PigScriptParser.java:320)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>>> GruntParser.java:194)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>>> GruntParser.java:170)
>>>>>>>>> at org.apache.pig.tools.grunt.**Grunt.run(Grunt.java:69)
>>>>>>>>> at org.apache.pig.Main.run(Main.**java:538)
>>>>>>>>> at org.apache.pig.Main.main(Main.**java:157)
>>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**
>>>>>>>>> invoke0(Native
>>>>>>>>> Method)
>>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>>>>>>>>> NativeMethodAccessorImpl.java:**57)
>>>>>>>>> at sun.reflect.**DelegatingMethodAccessorImpl.*
>>>>>>>>> *invoke(**
>>>>>>>>> DelegatingMethodAccessorImpl.**java:43)
>>>>>>>>> at java.lang.reflect.Method.**invoke(Method.java:606)
>>>>>>>>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:212)
>>>>>>>>> Caused by: java.io.IOException: Job terminated with anomalous
>>>>>>>>> status
>>>>>>>>> FAILED
>>>>>>>>> at org.apache.pig.PigServer.**
>>>>>>>>> openIterator(PigServer.java:**828)
>>>>>>>>> ... 12 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>
--
Regards,
Uttam Kumar
