[
https://issues.apache.org/jira/browse/FLINK-29425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song closed FLINK-29425.
--------------------------------
Fix Version/s: 1.16.0
Resolution: Fixed
- master (1.17): fe3bdcf0182ded35f1a8ac6eee4d7bf7acec832d
- release-1.16: 790434b9fb7296420c1ea15af0d640273776d0b0
> Hybrid full spilling strategy triggering spilling frequently
> ------------------------------------------------------------
>
> Key: FLINK-29425
> URL: https://issues.apache.org/jira/browse/FLINK-29425
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
> Affects Versions: 1.16.0
> Reporter: Weijie Guo
> Assignee: Weijie Guo
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> In hybrid shuffle mode, we have an internal config option
> 'DEFAULT_FULL_STRATEGY_NUM_BUFFERS_TRIGGER_SPILLED' to control spilling
> frequency in the full spilling strategy. Unfortunately, the default value(10)
> is too small. As a result, frequent disk spilling calls are observed in the
> TPC-DS test, which seriously affects performance. When we increase the value,
> the query performance is improved significantly. We should set a more
> reasonable default value, or adopt an adaptive strategy to determine the
> spilling frequency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
