[
https://issues.apache.org/jira/browse/IMPALA-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432631#comment-17432631
]
ASF subversion and git services commented on IMPALA-10873:
----------------------------------------------------------
Commit c127b6b1a72d83778378dc181d9328b4cb7dea9e in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c127b6b ]
IMPALA-10873: Push down EQUALS, IS NULL and IN-list predicate to ORC reader
This patch pushs down more kinds of predicates into the ORC reader,
including EQUALS, IN-list, and IS-NULL predicates to have more
improvements:
- EQUALS and IN-list predicates can be evaluated inside the ORC reader
with bloom filters in the ORC files.
- Comparing to scanning parquet that converting an IN-list predicate
into two binary predicates (i.e. LE and GE), the ORC reader can
leverage IN-list predicates to skip ORC RowGroups. E.g. a RowGroup
with int column 'x' in range [1, 100] will be skipped if we push down
predicate "x in (0, 101)".
- IS-NULL predicates (including IS-NOT-NULL) can also be used in the
ORC reader to skip RowGroups.
Implementation:
FE will collect these kinds of predicates into 'min_max_conjuncts' of
THdfsScanNode. To better reflect the meaning, 'min_max_conjuncts' is
renamed to 'stats_conjuncts'. Same for other related variable names.
Parquet scanner will only pick binary min-max conjuncts (i.e. LT, GT,
LE, and GE) to keep the existing behavior. ORC scanner will build
SearchArgument based on all these conjuncts.
Tests
* Add a new test table 'alltypessmall_bool_sorted' which has files
contiaining sorted bool values.
* Add test in orc-stats.test
Change-Id: Iaa89f080fe2e87d94fc8ea7f1be83e087fa34225
Reviewed-on: http://gerrit.cloudera.org:8080/17815
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Qifan Chen <[email protected]>
> Push down EQUALS, IS NULL and IN-list predicate to ORC reader
> -------------------------------------------------------------
>
> Key: IMPALA-10873
> URL: https://issues.apache.org/jira/browse/IMPALA-10873
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> IMPALA-6505 pushs down the min-max predicates into the ORC reader. Since
> ORC's SearchArguments also support IN-list predicates, we can consider
> pushing down IN-list and not IN-list predicates into it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
