[
https://issues.apache.org/jira/browse/HCATALOG-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise updated HCATALOG-347:
----------------------------------
Fix Version/s: 0.5
> Pig loads all partitions when it is specifically told not to load all
> ---------------------------------------------------------------------
>
> Key: HCATALOG-347
> URL: https://issues.apache.org/jira/browse/HCATALOG-347
> Project: HCatalog
> Issue Type: Bug
> Components: pig
> Affects Versions: 0.2
> Reporter: Viraj Bhat
> Assignee: Thomas Weise
> Fix For: 0.5
>
>
> I have Pig script of this nature. It accesses a partitioned table,
> partitioned on gridname and dt (datestamp)
> {code}
> A = LOAD 'mytable' USING org.apache.hcatalog.pig.HCatLoader();
> B = FILTER A BY gridname=='XY' and dt != '2012_03_21';
> C = foreach B generate job_id, user;
> store C into '/user/viraj/test/XY' using PigStorage();
> {code}
> I use this as some partitions of the table have not been populated.
> I get an error:
> Backend error message during job submission
> {quote}
> -------------------------------------------
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path
> does not exist:
> hdfs://namenode/warehouse/database_confs/gridname=XY/dt=2012_03_21
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:282)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
> at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1082)
> {quote}
> I suspect that the filter clause is not pushed up.
> Regards
> Viraj
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
