Weijie Guo created FLINK-29425: ---------------------------------- Summary: Hybrid full spilling strategy triggering spilling frequently Key: FLINK-29425 URL: https://issues.apache.org/jira/browse/FLINK-29425 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.16.0 Reporter: Weijie Guo
In hybrid shuffle mode, we have an internal config option 'DEFAULT_FULL_STRATEGY_NUM_BUFFERS_TRIGGER_SPILLED' to control spilling frequency in the full spilling strategy. Unfortunately, the default value(10) is too small. As a result, frequent disk spilling calls are observed in the TPC-DS test, which seriously affects performance. When we increase the value, the query performance is improved significantly. We should set a more reasonable default value, or adopt an adaptive strategy to determine the spilling frequency. -- This message was sent by Atlassian Jira (v8.20.10#820010)