Gopal V created HIVE-16852:
------------------------------
Summary: PTF: RANK() re-evaluates order predicates on the reducer
Key: HIVE-16852
URL: https://issues.apache.org/jira/browse/HIVE-16852
Project: Hive
Issue Type: Bug
Components: Physical Optimizer
Affects Versions: 2.1.1, 3.0.0
Reporter: Gopal V
{code}
explain select ss_item_sk, rank() over(order by cast(ss_list_price as
decimal(38,10))) as r , ss_list_price from store_sales;
STAGE PLANS:
Stage: Stage-1
Tez
DagId: root_20170608015140_7b0debb9-b14b-4150-b004-9743c6127392:3
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE)
DagName:
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: store_sales
Statistics: Num rows: 28800426268 Data size: 450435120648
Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: 0 (type: int), CAST( ss_list_price AS
decimal(38,10)) (type: decimal(38,10))
sort order: ++
Map-reduce partition columns: 0 (type: int)
Statistics: Num rows: 28800426268 Data size: 450435120648
Basic stats: COMPLETE Column stats: COMPLETE
value expressions: ss_item_sk (type: bigint), ss_list_price
(type: double)
Execution mode: vectorized, llap
LLAP IO: all inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
Select Operator
expressions: VALUE._col1 (type: bigint), VALUE._col11 (type:
double)
outputColumnNames: _col1, _col11
Statistics: Num rows: 28800426268 Data size: 8399352770616
Basic stats: COMPLETE Column stats: COMPLETE
PTF Operator
Function definitions:
Input definition
input alias: ptf_0
output shape: _col1: bigint, _col11: double
type: WINDOWING
Windowing table definition
input alias: ptf_1
name: windowingtablefunction
order by: CAST( _col11 AS decimal(38,10)) ASC NULLS
FIRST
partition by: 0
raw input shape:
window functions:
window function definition
alias: rank_window_0
arguments: CAST( _col11 AS decimal(38,10))
name: rank
window function: GenericUDAFRankEvaluator
window frame: PRECEDING(MAX)~FOLLOWING(MAX)
isPivotResult: true
Statistics: Num rows: 28800426268 Data size: 8399352770616
Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col1 (type: bigint), rank_window_0 (type:
int), _col11 (type: double)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 28800426268 Data size: 565636825720
Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
Statistics: Num rows: 28800426268 Data size: 565636825720
Basic stats: COMPLETE Column stats: COMPLETE
table:
input format:
org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde:
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
{code}
This forces the Decimal cast to be evaluated ~2x - once to produce the KEY
expression and once within the window function.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
