[
https://issues.apache.org/jira/browse/HIVE-22238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961081#comment-16961081
]
Zoltan Haindrich commented on HIVE-22238:
-----------------------------------------
written some tests to check how well a particular solution works
I've added 2 small "tweaks":
* column stats are marked if they are used in a filter expr
* "is null" filters are not considered as affected by the filter if the number
of nulls is 0 in the columnstat
there is one test which doesnt pass with accurate estimates on the second
join; because the first join has a .01 selectivity from the 2 0.1 branches; but
min is used during estimation so it kept as .1 - I think this is okay.
I've uploaded a new patch to run hiveqa - will clean up the patch for the next
run
> PK/FK selectivity estimation underscales estimations
> ----------------------------------------------------
>
> Key: HIVE-22238
> URL: https://issues.apache.org/jira/browse/HIVE-22238
> Project: Hive
> Issue Type: Bug
> Components: Statistics
> Reporter: Zoltan Haindrich
> Assignee: Zoltan Haindrich
> Priority: Major
> Attachments: HIVE-22238.01.patch, HIVE-22238.02.patch,
> HIVE-22238.03.patch, HIVE-22238.04.patch
>
>
> at [this
> point|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2182]
> the parent operators rownum is scaled according to pkfkselectivity
> however [pkfkselectivity is
> computed|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2157]
> on a whole subtree.
> Scaling it by that amount will count in estimation already used when
> parentstats was calculated...so depending on the number of upstream joins -
> this may lead to severe underestimations
> what happened was:
> * optimization was able to push the filter to the other side of the join
> * as a result the incoming data was already filtered
> * scaling down by the PK selectiviy - was actually already there...but a new
> "scaling" happened
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
