Has anyone tried using PigServer with Hadoop 0.23 and Yarn-specifically the
CDH4b1 release?
Running Pig scripts through Grunt works without hiccup but I have a utility
program that runs commands through PigServer. When running that way, I run
into two problems.
The first, which I can hack around, is that I need to pass in to PigServer the
hbase configuration. When I do this with
ConfigurationUtil.toProperties(hbaseConfig), the new properties get passed
in-not the deprecated ones. Later on down the road, Pig looks for
fs.default.name which is the deprecated property for fs.defaultfs and throws
[1] The hack is to back map the fs.default.name property to the value of
fs.defaultfs.
That brings me to the second problem which I haven't been able to hack around
:( When PigServer runs, it throws the exception[2]. Poking through the code,
there is no longer an org.apache.hadoop.mapred.jobcontrol.JobControl.addJob()
method distributed with hadoop 0.23-only an
org.apache.hadoop.mapred.jobcontrol.JobControl.addJobs() method-which also
appears to be deprecated.
Again, Grunt works fine-just PigServer shows these problems. Has anyone
resolved this? Or should I log a bug?
Thanks!
[1]
java.lang.NullPointerException
at java.net.URI$Parser.parse(URI.java:3003)
at java.net.URI.<init>(URI.java:578)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.setHdfsServers(QueryParser.java:485)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:1618)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1366)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:1013)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:800)
at
org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1612)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1562)
at org.apache.pig.PigServer.registerQuery(PigServer.java:534)
at org.apache.pig.PigServer.registerQuery(PigServer.java:558)
at com.cerner.kepler.pig.console.KeyDump.dumpKeys(KeyDump.java:227)
[2]
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Lja
va/lang/String;
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:258)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:382)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1209)
at org.apache.pig.PigServer.storeEx(PigServer.java:885)
at org.apache.pig.PigServer.store(PigServer.java:827)
at com.cerner.kepler.pig.console.KeyDump.dumpKeys(KeyDump.java:231)
Brian Tieman
Software Architect
[email protected]<mailto:[email protected]>
www.cerner.com
----------------------------------------------------------------------
CONFIDENTIALITY NOTICE This message and any included attachments are from
Cerner Corporation and are intended only for the addressee. The information
contained in this message is confidential and may constitute inside or
non-public information under international, federal, or state securities laws.
Unauthorized forwarding, printing, copying, distribution, or use of such
information is strictly prohibited and may be unlawful. If you are not the
addressee, please promptly delete this message and notify the sender of the
delivery error by e-mail or you may call Cerner's corporate offices in Kansas
City, Missouri, U.S.A at (+1) (816)221-1024.
