[
https://issues.apache.org/jira/browse/PIG-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272867#comment-13272867
]
Bill Graham commented on PIG-2693:
----------------------------------
When {{HCatalog}} blows up, the stack trace looks like this:
{noformat}
2012-05-05 00:43:09,684 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Couldn't get statistics from LoadFunc:
com.twitter.twadoop.dal.pig.DALPigLoader@2a2096d7
java.io.IOException: java.lang.NullPointerException
at org.apache.hcatalog.pig.HCatLoader.getStatistics(HCatLoader.java:194)
at
com.twitter.twadoop.dal.pig.DALPigLoader.getStatistics(DALPigLoader.java:135)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getInputSizeFromLoader(JobControlCompiler.java:866)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getInputSize(JobControlCompiler.java:824)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:805)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:745)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:378)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:264)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:149)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1265)
...
{noformat}
> LoadFunc.setLocation should be called before LoadMetadata.getStatistics
> ------------------------------------------------------------------------
>
> Key: PIG-2693
> URL: https://issues.apache.org/jira/browse/PIG-2693
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Assignee: Bill Graham
>
> We ran into a bug with Pig/HCatalog integration on the trunk. The issue is
> that {{JobControlCompiler}} calls the {{adjustNumReducers}} method just
> before it calls {{setLocation}} on all of the {{LoadFuncs}}. This causes
> problems, since some loaders (i.e., {{HCatLoader}}) need {{setLocation}} to
> be called before it can respond to {{getStatistics}} with it's data size.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
