[
https://issues.apache.org/jira/browse/HIVE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235107#comment-16235107
]
Sahil Takiar commented on HIVE-17958:
-------------------------------------
So looks like {{RedundantDynamicPruningConditionsRemoval}} has some bugs in it.
It completely disables DPP for the following query:
{code}
EXPLAIN SELECT count(*) FROM partitioned_table1 WHERE
partitioned_table1.part_col IN (
SELECT regular_table1.col1 FROM regular_table1 JOIN partitioned_table2 ON
regular_table1.col1 = partitioned_table2.part_col AND partitioned_table2.col >
3 AND regular_table1.col1 > 1)
{code}
> spark_dynamic_partition_pruning.q fails when
> hive.tez.dynamic.semijoin.reduction is false
> -----------------------------------------------------------------------------------------
>
> Key: HIVE-17958
> URL: https://issues.apache.org/jira/browse/HIVE-17958
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
>
> Looks like {{RedundantDynamicPruningConditionsRemoval}} causes DPP to be
> disabled in a few cases (not sure why). When
> {{hive.tez.dynamic.semijoin.reduction}} is {{true}} (the default), then this
> rule is disabled so the normal tests don't hit this issue.
> But when I disable {{hive.tez.dynamic.semijoin.reduction}} then the following
> query no longer fully triggers DPP:
> {code}
> EXPLAIN select count(*) from srcpart join srcpart_date on (srcpart.ds =
> srcpart_date.ds) join srcpart_hour on (srcpart.hr = srcpart_hour.hr)
> 5777 where srcpart_date.`date` = '2008-04-08' and srcpart_hour.hour = 11 and
> srcpart.hr = 11
> {code}
> There should be two DPP sinks, but when the config is set to false, there is
> only one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
