[jira] [Work logged] (HIVE-24165) CBO: Query fails after multiple count distinct rewrite
[
https://issues.apache.org/jira/browse/HIVE-24165?focusedWorklogId=504048&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-504048
]
ASF GitHub Bot logged work on HIVE-24165:
-
Author: ASF GitHub Bot
Created on: 23/Oct/20 06:42
Start Date: 23/Oct/20 06:42
Worklog Time Spent: 10m
Work Description: loudongfeng closed pull request #1597:
URL: https://github.com/apache/hive/pull/1597
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 504048)
Time Spent: 20m (was: 10m)
> CBO: Query fails after multiple count distinct rewrite
> ---
>
> Key: HIVE-24165
> URL: https://issues.apache.org/jira/browse/HIVE-24165
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-24165.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> One way to reproduce:
>
> {code:sql}
> CREATE TABLE test(
> `device_id` string,
> `level` string,
> `site_id` string,
> `user_id` string,
> `first_date` string,
> `last_date` string,
> `dt` string) ;
> set hive.execution.engine=tez;
> set hive.optimize.distinct.rewrite=true;
> set hive.cli.print.header=true;
> select
> dt,
> site_id,
> count(DISTINCT t1.device_id) as device_tol_cnt,
> count(DISTINCT case when t1.first_date='2020-09-15' then t1.device_id else
> null end) as device_add_cnt
> from test t1 where dt='2020-09-15'
> group by
> dt,
> site_id
> ;
> {code}
>
> Error log:
> {code:java}
> Exception in thread "main" java.lang.AssertionError: Cannot add expression of
> different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" $f2, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" $f3, BIGINT $f2_0, BIGINT $f3_0) NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" $f2, BIGINT $f3, BIGINT $f2_0, BIGINT
> $f3_0) NOT NULL
> set is rel#85:HiveAggregate.HIVE.[](input=HepRelVertex#84,group={2,
> 3},agg#0=count($0),agg#1=count($1))
> expression is HiveProject#95
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:411)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:57)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at
> org.apache.calcite.rel.rules.AggregateProjectPullUpConstantsRule.onMatch(AggregateProjectPullUpConstantsRule.java:186)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:317)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:415)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:280)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:211)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:198)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:2273)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:2002)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1709)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1609)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:154)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:111)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1414)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1430)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:450)
> at
> org.apache.had
[jira] [Work logged] (HIVE-24165) CBO: Query fails after multiple count distinct rewrite
[
https://issues.apache.org/jira/browse/HIVE-24165?focusedWorklogId=503704&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-503704
]
ASF GitHub Bot logged work on HIVE-24165:
-
Author: ASF GitHub Bot
Created on: 22/Oct/20 12:36
Start Date: 22/Oct/20 12:36
Worklog Time Spent: 10m
Work Description: loudongfeng opened a new pull request #1597:
URL: https://github.com/apache/hive/pull/1597
### What changes were proposed in this pull request?
Keep Aggregate's groupSet in order during multiple distinct rewrite.
### Why are the changes needed?
Fix column mismatch issue between HiveExpandDistinctAggregatesRule and
AggregateProjectPullUpConstantsRule
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Test by qtests
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 503704)
Remaining Estimate: 0h
Time Spent: 10m
> CBO: Query fails after multiple count distinct rewrite
> ---
>
> Key: HIVE-24165
> URL: https://issues.apache.org/jira/browse/HIVE-24165
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Major
> Attachments: HIVE-24165.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> One way to reproduce:
>
> {code:sql}
> CREATE TABLE test(
> `device_id` string,
> `level` string,
> `site_id` string,
> `user_id` string,
> `first_date` string,
> `last_date` string,
> `dt` string) ;
> set hive.execution.engine=tez;
> set hive.optimize.distinct.rewrite=true;
> set hive.cli.print.header=true;
> select
> dt,
> site_id,
> count(DISTINCT t1.device_id) as device_tol_cnt,
> count(DISTINCT case when t1.first_date='2020-09-15' then t1.device_id else
> null end) as device_add_cnt
> from test t1 where dt='2020-09-15'
> group by
> dt,
> site_id
> ;
> {code}
>
> Error log:
> {code:java}
> Exception in thread "main" java.lang.AssertionError: Cannot add expression of
> different type to set:
> set type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" $f2, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" $f3, BIGINT $f2_0, BIGINT $f3_0) NOT NULL
> expression type is RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> COLLATE "ISO-8859-1$en_US$primary" $f2, BIGINT $f3, BIGINT $f2_0, BIGINT
> $f3_0) NOT NULL
> set is rel#85:HiveAggregate.HIVE.[](input=HepRelVertex#84,group={2,
> 3},agg#0=count($0),agg#1=count($1))
> expression is HiveProject#95
> at
> org.apache.calcite.plan.RelOptUtil.verifyTypeEquivalence(RelOptUtil.java:411)
> at
> org.apache.calcite.plan.hep.HepRuleCall.transformTo(HepRuleCall.java:57)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at
> org.apache.calcite.rel.rules.AggregateProjectPullUpConstantsRule.onMatch(AggregateProjectPullUpConstantsRule.java:186)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:317)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:415)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:280)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:211)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:198)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.hepPlan(CalcitePlanner.java:2273)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:2002)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1709)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1609)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:118)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1052)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:
