Pig loads all partitions when it is specifically told not to load all
---------------------------------------------------------------------
Key: HCATALOG-347
URL: https://issues.apache.org/jira/browse/HCATALOG-347
Project: HCatalog
Issue Type: Bug
Components: pig
Affects Versions: 0.2
Reporter: Viraj Bhat
I have Pig script of this nature. It accesses a partitioned table, partitioned
on gridname and dt (datestamp)
{code}
A = LOAD 'mytable' USING org.apache.hcatalog.pig.HCatLoader();
B = FILTER A BY gridname=='XY' and dt != '2012_03_21';
C = foreach B generate job_id, user;
store C into '/user/viraj/test/XY' using PigStorage();
{code}
I use this as some partitions of the table have not been populated.
I get an error:
Backend error message during job submission
{quote}
-------------------------------------------
org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path
does not exist:
hdfs://namenode/warehouse/database_confs/gridname=XY/dt=2012_03_21
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:282)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1082)
{quote}
I suspect that the filter clause is not pushed up.
Regards
Viraj
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
