predicate on partitioning column is ignored in many places
----------------------------------------------------------
Key: HIVE-218
URL: https://issues.apache.org/jira/browse/HIVE-218
Project: Hadoop Hive
Issue Type: Bug
Components: Query Processor
Reporter: Joydeep Sen Sarma
Priority: Blocker
We tried two queries yesterday that bought up several problems:
1. predicate on partitioning column within a join clause was ignored:
FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM
yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT
c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx AND
e.ds=2009-01-05) INSERT OVERWRITE TABLE ...
the plan tried to scan *all* partitions!
2. predicate on partitioning clause inside insert clause was ignored (we took
the previous query and moved the partition filter to the insert statement)
FROM (FROM xxx a SELECT a.xx, a.yy, a.ds WHERE a.ds=2009-01-05 UNION ALL FROM
yyy SELECT b.xx, b.yy, b.ds WHERE b.ds=2009-01-05 UNION ALL FROM zzz c SELECT
c.xx, c.yy, c.ds WHERE c.ds=2009-01-05) d JOIN aaa e ON (d.xx=e.xx ) INSERT
OVERWRITE TABLE ... WHERE e.ds=2009-01-05;
the plan again tried to scan *all* partitions
the really bad thing is that we were able to detect this problem only because
of metastore inconsistencies - otherwise - we would have merrily scanned all
the data. This is really critical to get fixed - because this means that we may
actually be scanning tons of unnecessary data in production.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
