feiniaofeiafei commented on PR #64892:
URL: https://github.com/apache/doris/pull/64892#issuecomment-5337078543
Currently, when primary-key/foreign-key constraints are absent, TPC-DS
queries 4 and 11 experience a performance regression.
The regression occurs because
org.apache.doris.nereids.properties.RequestPropertyDeriver#shouldUseParent
rejects ss_customer_sk, a column derived from a join key, as the aggregate
shuffle key.
The table contains approximately 2.87 × 10^9 rows. The column contains about
6.7 × 10^7 null values and has an NDV of approximately 10^7. Although null
occurs much more frequently than any individual non-null value, nulls account
for only about 2.3% of the total rows. Therefore, using this column as the
shuffle key should be acceptable.
However, the current implementation of shouldUseParent rejects this column,
causing the optimizer to choose a broadcast join for the relatively large
customer table. Broadcasting this table results in a significant performance
regression.
The suggested fix is to use the following method to determine whether the
data is skewed:
if (StatisticsUtil.isBalancedAllowUnknownHotValues(
columnStatistic, instanceNum, 0.05, inputStatistics.getRowCount())) {
continue;
}
public static boolean isBalancedAllowUnknownHotValues(
ColumnStatistic columnStatistic, int instanceNum,
double minRatio, double rowCount) {
double ndv = columnStatistic.ndv;
return ndv > instanceNum * AggregateUtils.NDV_INSTANCE_BALANCE_MULTIPLIER
&& !hasSignificantHotValues(
columnStatistic, minRatio, rowCount, false);
}
Instead of using:
public static boolean isHotValueWithOriginalThreshold(
double ratio, double ndv) {
return ratio >= SessionVariable.getHotValueThreshold()
|| ratio * ndv >= SessionVariable.getSkewValueThreshold();
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
