This is an automated email from the ASF dual-hosted git repository.
gavinchou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 28ff5b2c81f [fix](load) reduce S3 upload queue backpressure threshold
(#67058)
28ff5b2c81f is described below
commit 28ff5b2c81f0c92e4264f5f0e6523e0879722189
Author: hui lai <[email protected]>
AuthorDate: Mon Aug 24 21:41:41 2026 +0800
[fix](load) reduce S3 upload queue backpressure threshold (#67058)
Related PR #66847 changed S3-backed cloud loads to use the shared S3
file upload thread-pool queue as the backpressure signal. However, its
queue threshold of 100 was too high for this pipeline and introduced a
severe load-performance regression.
Once S3 upload tasks start accumulating in the queue, the upload side is
already unable to keep up with producers. Allowing writers to continue
flushing does not increase S3 upload throughput; it only leaves more
pending upload data in memory. Backpressure should therefore be applied
early when the S3 upload queue starts building up.
During the DORIS-28052 benchmark, the queue repeatedly overshot the
threshold of 100 and reached about 2,100 queued tasks per BE. This PR
lowers `kS3QueueBusyThreshold` from 100 to 10. The benchmark results
below show that earlier backpressure prevents excessive queue buildup
and recovers the load-performance regression.
Benchmark results (SF1000 load):
#### 1. Baseline
| Dataset | Load time |
|---|---:|
| `ssb_flat` | 1,530.10 s |
| `tpcds` | 1,414.04 s |
| `ssb` | 781.46 s |
| `tpch` | 1,382.67 s |
#### 2. Regression before this PR
| Dataset | Load time | Regression vs. baseline |
|---|---:|---:|
| `ssb_flat` | 3,421.53 s | +123.61% |
| `tpcds` | 2,678.30 s | +89.41% |
| `ssb` | 962.48 s | +23.16% |
| `tpch` | 2,034.80 s | +47.16% |
#### 3. Result after lowering the threshold from 100 to 10
(`f08e45192a`)
| Dataset | Load time | Improvement by this PR | Final vs. baseline |
|---|---:|---:|---:|
| `ssb_flat` | 1,364.847 s | 60.11% faster | -10.80% |
| `tpcds` | 1,405.438 s | 47.52% faster | -0.61% |
| `ssb` | 781.869 s | 18.77% faster | +0.05% |
| `tpch` | 1,443.310 s | 29.07% faster | +4.39% |
Lowering the backpressure threshold from 100 to 10 recovers the load
regression across all four workloads. Compared with the regressed
result, load time improves by 18.77% to 60.11%; the final results are
within 4.39% of the baseline.
This remains a soft backpressure trigger rather than a hard queue-size
limit.
---
be/src/storage/adaptive_thread_pool_controller.h | 2 +-
be/test/storage/adaptive_thread_pool_controller_test.cpp | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/storage/adaptive_thread_pool_controller.h
b/be/src/storage/adaptive_thread_pool_controller.h
index 0598531268c..ae78b7883dd 100644
--- a/be/src/storage/adaptive_thread_pool_controller.h
+++ b/be/src/storage/adaptive_thread_pool_controller.h
@@ -80,7 +80,7 @@ public:
static constexpr int kQueueThreshold = 10;
static constexpr int kIOBusyThresholdPercent = 90;
static constexpr int kCPUBusyThresholdPercent = 90;
- static constexpr int kS3QueueBusyThreshold = 100;
+ static constexpr int kS3QueueBusyThreshold = 10;
AdaptiveThreadPoolController() = default;
~AdaptiveThreadPoolController() { stop(); }
diff --git a/be/test/storage/adaptive_thread_pool_controller_test.cpp
b/be/test/storage/adaptive_thread_pool_controller_test.cpp
index 6c4e42fea05..8e3b34841f8 100644
--- a/be/test/storage/adaptive_thread_pool_controller_test.cpp
+++ b/be/test/storage/adaptive_thread_pool_controller_test.cpp
@@ -226,7 +226,7 @@ TEST_F(AdaptiveThreadPoolControllerTest, TestConstants) {
EXPECT_EQ(AdaptiveThreadPoolController::kQueueThreshold, 10);
EXPECT_EQ(AdaptiveThreadPoolController::kIOBusyThresholdPercent, 90);
EXPECT_EQ(AdaptiveThreadPoolController::kCPUBusyThresholdPercent, 90);
- EXPECT_EQ(AdaptiveThreadPoolController::kS3QueueBusyThreshold, 100);
+ EXPECT_EQ(AdaptiveThreadPoolController::kS3QueueBusyThreshold, 10);
}
// Test add after controller is already running
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
