[ https://issues.apache.org/jira/browse/HIVE-18779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Sun resolved HIVE-18779. ------------------------------ Resolution: Not A Problem sorry , i did not notice the note in the hive configuration wiki : Note: Turn on *[hive.optimize.index.filter|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.optimize.index.filter]* as well to use file format specific indexes with PPD. > Hive does not enable ppd to underlying storage format by default > ---------------------------------------------------------------- > > Key: HIVE-18779 > URL: https://issues.apache.org/jira/browse/HIVE-18779 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer, Query Planning > Affects Versions: 1.2.1, 1.2.2, 2.3.2 > Environment: Hive 1.2.1 and also checked the latest version , it > still have this issue. > Reporter: Keith Sun > Priority: Major > Attachments: image-2018-02-22-20-29-41-589.png > > > *Issue :* Hive does not enable ppd to underlying storage format by default > even with hive.optimize.ppd/storage=true and the inputFormat is applicable > for fitler push down. > *How to re-produce :* > > {code:java} > CREATE TABLE MYDUAL (ID INT) stored as parquet; > insert overwrite table mydual ... > set hive.optimize.ppd=true > set hive.optimize.ppd.storage=true > explain select * from mydual where id =100; > //No filterExpr generated which will be utilized by Parquet InputFormat > STAGE PLANS: > Stage: Stage-0 > Fetch Operator > limit: -1 > Processor Tree: > TableScan > alias: mydual > Statistics: Num rows: 362 Data size: 362 Basic stats: COMPLETE Column stats: > NONE > Filter Operator > predicate: (id = 100) (type: boolean) > Statistics: Num rows: 181 Data size: 181 Basic stats: COMPLETE Column stats: > NONE > Select Operator > expressions: 100 (type: int) > //set hive.optimize.index.filter=true which is false by default. > //then we get the filterExpr which can be pushed down to parquet. > STAGE PLANS: Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: > TableScan alias: mydual filterExpr: (id = 100) (type: boolean) Statistics: > Num rows: 362 Data size: 362 Basic stats: COMPLETE Column stats: NONE Filter > Operator predicate: (id = 100) (type: boolean) Statistics: Num rows: 181 Data > size: 181 Basic stats: COMPLETE Column stats: NONE Select Operator > expressions: 100 (type: int) outputColumnNames: _col0 Statistics: Num rows: > 181 Data size: 181 Basic stats: COMPLETE Column stats: NONE ListSink > {code} > By checking the code of org.apache.hadoop.hive.ql.ppd.OpProcFactory: > I just found that to generate the filterExpr in the plan, we have to set : > hive.optimize.index.filter=true as a workaround, but this parameter is not > related to parquet input format as we does not have index at all. > > {code:java} > private static ExprNodeGenericFuncDesc pushFilterToStorageHandler( > TableScanOperator tableScanOp, > ExprNodeGenericFuncDesc originalPredicate, > OpWalkerInfo owi, > HiveConf hiveConf) { > TableScanDesc tableScanDesc = tableScanOp.getConf(); > Table tbl = tableScanDesc.getTableMetadata(); > if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTINDEXFILTER)) { > // attach the original predicate to the table scan operator for index > // optimizations that require the pushed predicate before pcr & later > // optimizations are applied > tableScanDesc.setFilterExpr(originalPredicate); > } > if (!tbl.isNonNative()) { > return originalPredicate; > } > HiveStorageHandler storageHandler = tbl.getStorageHandler(); > if (!(storageHandler instanceof HiveStoragePredicateHandler)) { > // The storage handler does not provide predicate decomposition > // support, so we'll implement the entire filter in Hive. However, > // we still provide the full predicate to the storage handler in > // case it wants to do any of its own prefiltering. > tableScanDesc.setFilterExpr(originalPredicate); > return originalPredicate; > } > HiveStoragePredicateHandler predicateHandler = > (HiveStoragePredicateHandler) storageHandler; > JobConf jobConf = new JobConf(owi.getParseContext().getConf()); > Utilities.setColumnNameList(jobConf, tableScanOp); > Utilities.setColumnTypeList(jobConf, tableScanOp); > Utilities.copyTableJobPropertiesToConf( > Utilities.getTableDesc(tbl), > jobConf); > {code} > Per my check , the "getFilterExpr" method of TableScanDesc is called below > places and > If hive always set the filterExpr, it may not cause trouble (chime in if i am > wrong). > !image-2018-02-22-20-29-41-589.png! > I could propose a pull request then. > !image-2018-02-22-20-23-07-732.png! > -- This message was sent by Atlassian JIRA (v7.6.3#76005)