[
https://issues.apache.org/jira/browse/HIVE-24080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
范宜臻 updated HIVE-24080:
-----------------------
Description:
{code:java}
drop table temp.test_cbo_fyz_right;
CREATE EXTERNAL TABLE temp.test_cbo_right(
`rid` string,
`dtm` string,
`map` map<string, string>,
`tags` array <string>
);drop table temp.test_cbo_fyz_left;
CREATE EXTERNAL TABLE temp.test_cbo_left(
`lid` string,
`dtm` string
)
explain select
b.tags
from temp.test_cbo_left a
left join temp.test_cbo_right b on a.lid = b.rid
where b.dtm = '20200814'
and a.dtm = '20200813'{code}
The Problem will occur in left join and select array type or map type column
in right table(reverse for right join or both for full join).
code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from
calcite and then convert it back to `optimizedAST`, the projection of b.tags
has been optimized to be RexCall => CAST(b.tags) in calcite code. Normally the
projection should be an RexInputRef, but in this case, calcite need cast `not
null` to `nullable`. However, TypeConverter in ASTConverter has not support
ARRAY and MAP.
NPE threw here:
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor
visitCall
{code:java}
HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null
ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text);
if (ht.args != null) {
for (String castArg : ht.args)
astBldr.add(HiveParser.Identifier, castArg);
}
{code}
The final result is cbo failed and skip CalcitePlan and cause partition pruning
failed.
when table is large, sql will explode.
I wonder which project should fix this problem
was:
{code:java}
drop table temp.test_cbo_fyz_right;
CREATE EXTERNAL TABLE temp.test_cbo_right(
`rid` string,
`dtm` string,
`map` map<string, string>,
`tags` array <string>
);drop table temp.test_cbo_fyz_left;
CREATE EXTERNAL TABLE temp.test_cbo_left(
`lid` string,
`dtm` string
)
explain select
b.tags
from temp.test_cbo_left a
left join temp.test_cbo_right b on a.lid = b.rid
where b.dtm = '20200814'
and a.dtm = '20200813'{code}
The Problem will occur in left join and select array type or map type column
in right table(reverse for right join or both for full join).
code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from
calcite and then convert it back to `optimizedAST`, the projection of b.tags
has been optimized to be RexCall => CAST(b.tags) in calcite code. Normally the
projection should be an RexInputRef, but in this case, calcite need cast `not
null` to `nullable`. However, TypeConverter in ASTConverter has not support
ARRAY and MAP.
NPE threw here:
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor
visitCall
{code:java}
HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null
ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text);
if (ht.args != null) {
for (String castArg : ht.args)
astBldr.add(HiveParser.Identifier, castArg);
}
{code}
The final result is cbo failed and skip CalcitePlan and cause partition pruning
failed.
when table is large, sql will explode.
I wonder which repo should fix this problem?
> CBO failed: Array and Map are not support when convert calcite types to hive
> types
> ----------------------------------------------------------------------------------
>
> Key: HIVE-24080
> URL: https://issues.apache.org/jira/browse/HIVE-24080
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
> Affects Versions: 3.0.0, 2.3.4
> Reporter: 范宜臻
> Assignee: Ashutosh Chauhan
> Priority: Critical
>
> {code:java}
> drop table temp.test_cbo_fyz_right;
> CREATE EXTERNAL TABLE temp.test_cbo_right(
> `rid` string,
> `dtm` string,
> `map` map<string, string>,
> `tags` array <string>
> );drop table temp.test_cbo_fyz_left;
> CREATE EXTERNAL TABLE temp.test_cbo_left(
> `lid` string,
> `dtm` string
> )
> explain select
> b.tags
> from temp.test_cbo_left a
> left join temp.test_cbo_right b on a.lid = b.rid
> where b.dtm = '20200814'
> and a.dtm = '20200813'{code}
>
> The Problem will occur in left join and select array type or map type column
> in right table(reverse for right join or both for full join).
> code will throw NPE when `calcitePlanner` generate `optimizedOptiqPlan` from
> calcite and then convert it back to `optimizedAST`, the projection of b.tags
> has been optimized to be RexCall => CAST(b.tags) in calcite code. Normally
> the projection should be an RexInputRef, but in this case, calcite need cast
> `not null` to `nullable`. However, TypeConverter in ASTConverter has not
> support ARRAY and MAP.
>
> NPE threw here:
> org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.RexVisitor
> visitCall
> {code:java}
> HiveToken ht = TypeConverter.hiveToken(call.getType()); // here ht is null
> ASTBuilder astBldr = ASTBuilder.construct(ht.type, ht.text);
> if (ht.args != null) {
> for (String castArg : ht.args)
> astBldr.add(HiveParser.Identifier, castArg);
> }
> {code}
> The final result is cbo failed and skip CalcitePlan and cause partition
> pruning failed.
> when table is large, sql will explode.
>
> I wonder which project should fix this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
