godfrey he commented on FLINK-15004:

[~ykt836], yes, `ndv` had been considered before: if `ndv` is unknown, the 
planner will choose two-phase aggregate. 
`DistinctRowCount` metadata handler could return null which means unknown, 
while `RowCount` metadata handler always returns primitive type and planner 
does not know whether the inputs have real row count or just use default value. 
the issue mainly solves the following scenarios: `ndv` is known, while row 
count is unknown.

> Choose two-phase Aggregate if the statistics is unknown
> -------------------------------------------------------
>                 Key: FLINK-15004
>                 URL: https://issues.apache.org/jira/browse/FLINK-15004
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table SQL / Planner
>    Affects Versions: 1.9.0, 1.9.1
>            Reporter: godfrey he
>            Assignee: godfrey he
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.10.0
>          Time Spent: 10m
>  Remaining Estimate: 0h
> Currently, blink planner will use default rowCount value (defined in 
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is 
> unknown, and maybe choose one-phase Aggregate. The job will hang if the data 
> is skewed. So It's better to use two-phase Aggregate for execution stability 
> if the statistics is unknown.

This message was sent by Atlassian Jira

Reply via email to