[ 
https://issues.apache.org/jira/browse/DRILL-4665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336911#comment-15336911
 ] 

Gautam Kumar Parai commented on DRILL-4665:
-------------------------------------------

I have created the pull request https://github.com/apache/drill/pull/526 
[~amansinha100] could you please take a look? Thanks!



> Partition pruning not working for hive partitioned table with 'LIKE' and '=' 
> filter
> -----------------------------------------------------------------------------------
>
>                 Key: DRILL-4665
>                 URL: https://issues.apache.org/jira/browse/DRILL-4665
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Query Planning & Optimization, Storage - Hive
>    Affects Versions: 1.6.0
>            Reporter: Chun Chang
>            Assignee: Gautam Kumar Parai
>
> This problem was initially reported by Shankar Mane 
> <shankar.m...@games24x7.com>
> > Problem:
> >
> > 1. In drill, we are using hive partition table. But explain plan (same
> > query) for like and = operator differs and used all partitions in case of
> > like operator.
> > 2. If you see below drill explain plans: Like operator uses *all*
> > partitions where
> > = operator uses *only* partition filtered by log_date condition.
> I reproduced the reported issue. I have a partitioned hive external table 
> with the following schema:
> {noformat}
> create external table if not exists lineitem_partitioned (
>     l_orderkey bigint,
>     l_partkey bigint,
>     l_suppkey bigint,
>     l_linenumber bigint,
>     l_quantity double,
>     l_extendedprice double,
>     l_discount double,
>     l_tax double,
>     l_returnflag string,
>     l_linestatus string,
>     l_shipdate string,
>     l_commitdate string,
>     l_receiptdate string,
>     l_shipinstruct string,
>     l_shipmode string,
>     l_comment string
> )
> partitioned by (year int, month int, day int)
> STORED AS PARQUET
> LOCATION '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=1) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/1';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=2) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=3) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/3';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=4) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/4';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=5) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/5';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=6) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/6';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=7) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/7';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=8) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/8';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=9) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/9';
> alter table lineitem_partitioned add partition(year=2015, month=1, day=10) 
> location  
> '/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/10';
> {noformat}
> Without 'LIKE', drill plans right:
> {noformat}
> 0: jdbc:drill:schema=dfs.drillTestDir> explain plan for select l_shipdate, 
> l_linestatus, l_shipinstruct, `day` from lineitem_partitioned_db2k_2_999 
> where `day` = 2 limit 10;
> +------+------+
> | text | json |
> +------+------+
> | 00-00    Screen
> 00-01      Project(l_shipdate=[$0], l_linestatus=[$1], l_shipinstruct=[$2], 
> day=[$3])
> 00-02        SelectionVectorRemover
> 00-03          Limit(fetch=[10])
> 00-04            Limit(fetch=[10])
> 00-05              Project(l_shipdate=[$1], l_linestatus=[$0], 
> l_shipinstruct=[$2], day=[$3])
> 00-06                Scan(groupscan=[HiveScan [table=Table(dbName:md815_db2k, 
> tableName:lineitem_partitioned_db2k_2_999), columns=[`l_linestatus`, 
> `l_shipdate`, `l_shipinstruct`, `day`], numPartitions=1, partitions= 
> [Partition(values:[2015, 1, 2])], 
> inputDirectories=[maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2]]])
> {noformat}
> With 'LIKE', pruning is not happening:
> {noformat}
> 0: jdbc:drill:schema=dfs.drillTestDir> explain plan for select l_shipdate, 
> l_linestatus, l_shipinstruct, `day` from lineitem_partitioned_db2k_2_999 
> where `day` = 2 and l_shipinstruct like '%BACK%' limit 10;
> +------+------+
> | text | json |
> +------+------+
> | 00-00    Screen
> 00-01      Project(l_shipdate=[$0], l_linestatus=[$1], l_shipinstruct=[$2], 
> day=[$3])
> 00-02        SelectionVectorRemover
> 00-03          Limit(fetch=[10])
> 00-04            Limit(fetch=[10])
> 00-05              Project(l_shipdate=[$1], l_linestatus=[$0], 
> l_shipinstruct=[$2], day=[$3])
> 00-06                SelectionVectorRemover
> 00-07                  Filter(condition=[AND(=($3, 2), LIKE($2, '%BACK%'))])
> 00-08                    Scan(groupscan=[HiveScan 
> [table=Table(dbName:md815_db2k, tableName:lineitem_partitioned_db2k_2_999), 
> columns=[`l_linestatus`, `l_shipdate`, `l_shipinstruct`, `day`], 
> numPartitions=10, partitions= [Partition(values:[2015, 1, 1]), 
> Partition(values:[2015, 1, 10]), Partition(values:[2015, 1, 2]), 
> Partition(values:[2015, 1, 3]), Partition(values:[2015, 1, 4]), 
> Partition(values:[2015, 1, 5]), Partition(values:[2015, 1, 6]), 
> Partition(values:[2015, 1, 7]), Partition(values:[2015, 1, 8]), 
> Partition(values:[2015, 1, 9])], 
> inputDirectories=[maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/1,
>  
> maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/10, 
> maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/2, 
> maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/3, 
> maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/4, 
> maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/5, 
> maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/6, 
> maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/7, 
> maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/8, 
> maprfs:/drill/testdata/tpch100_dir_partitioned_50000files/lineitem/2015/1/9]]])
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to