CalvinKirs commented on PR #70004: URL: https://github.com/apache/doris/pull/70004#issuecomment-6598787097
## branch-4.1 backport validation and performance Validated head: `5b90e691fce12665de078bb7b944357dfdc2ac72`. Baseline: `dd72933ec78a8a4daf3e1d204a6ea96d6443ada7` on branch-4.1, before this backport. Both sides use the same **BE 4.1.3** package (`doris-4.1.3-rc02-7126cf65d96`). ### Test conditions - Hive tables through a local Docker HMS, with real OSS and a dedicated HDFS fixture. Each partition contains 100 small Parquet files. The 100-, 1,000-, and 10,000-partition tables contain 10,000, 100,000, and 1,000,000 files respectively. - One serial query at a time. Both versions use one file-listing worker and one scheduling worker. Remaining relevant settings use branch-4.1 defaults, including the 1,024-partition threshold; the candidate's producer-parallelism setting stays at its default. - Same machine, CPU allocation, 4 GiB FE heap, and 6 GiB BE limit on both sides. An initial 3 GiB BE series was discarded after a baseline OOM; the reported comparison was rerun with 6 GiB on both sides. - Main comparison order: candidate / baseline / baseline / candidate, with two fresh FE processes per version. Each scenario has **20 measured samples per version**, following warm-up. All reported times are median SQL wall time in milliseconds. - "Fresh catalog" means a newly created FE catalog. It does not imply cold OSS service caches or cold OS caches. "Warm" means repeated queries after warm-up in the same catalog. Native listing workers and BE readers were checked for idleness around queries. - Performance timing runs use normal binaries. Resource/SDK observation is performed separately from timing. Here `t100`, `t1000`, and `t10000` are shorthand for the three table sizes. LIMIT SQL is `SELECT * FROM tN LIMIT 10`; full-scan SQL is `SELECT SUM(id) FROM t100`; filtered full-scan SQL is `SELECT SUM(id) FROM t10000 WHERE p < 100`. The two SUM scenarios read the same 100 partition directories / 10,000 physical files, verified through HMS. ### Main performance comparison Change = `(after / before - 1) * 100%`; a negative value means lower latency. | Storage | Scenario | Before (ms) | After (ms) | Change | |---|---|---:|---:|---:| | OSS | 100 partitions, LIMIT 10, fresh catalog | 1,606.6 | 1,643.7 | +2.3% | | OSS | 100 partitions, LIMIT 10, warm | 231.6 | 223.0 | -3.7% | | OSS | 1,000 partitions, LIMIT 10, fresh catalog | 14,746.0 | 14,918.1 | +1.2% | | OSS | 1,000 partitions, LIMIT 10, warm | 2,218.3 | 2,200.7 | -0.8% | | OSS | 10,000 partitions, LIMIT 10, fresh catalog | 1,087.5 | 584.1 | **-46.3%** | | OSS | 10,000 partitions, LIMIT 10, warm | 190.7 | 113.6 | **-40.5%** | | OSS | 100 partitions, SUM | 1,397.4 | 1,392.9 | -0.3% | | OSS | 10,000 partitions, SUM with p < 100 | 1,390.3 | 1,441.3 | +3.7% | | HDFS | 100 partitions, LIMIT 10, fresh catalog | 545.6 | 553.2 | +1.4% | | HDFS | 100 partitions, LIMIT 10, warm | 235.1 | 234.5 | -0.3% | | HDFS | 1,000 partitions, LIMIT 10, fresh catalog | 5,327.1 | 5,262.6 | -1.2% | | HDFS | 1,000 partitions, LIMIT 10, warm | 2,173.4 | 2,146.1 | -1.3% | | HDFS | 10,000 partitions, LIMIT 10, fresh catalog | 789.3 | 223.8 | **-71.6%** | | HDFS | 10,000 partitions, LIMIT 10, warm | 132.3 | 42.6 | **-67.8%** | | HDFS | 100 partitions, SUM | 1,187.4 | 1,086.8 | -8.5% | | HDFS | 10,000 partitions, SUM with p < 100 | 1,002.5 | 1,044.8 | +4.2% | The 10,000-partition LIMIT improvement reproduced in both execution orders for both storage systems. Smaller LIMIT scenarios did not show a stable improvement or regression across orders. These sizes remain below the existing 1,024-partition threshold, so the results should not be extrapolated as a uniform speedup for every table size. ### Additional confirmation of the slower filtered-SUM cases The main series showed roughly 4% higher medians for filtered SUM. These cases were repeated with **60 measured samples per version per scenario**, baseline / candidate / candidate / baseline process order, and alternating full-SUM / filtered-SUM query order. These are separate samples from the main table. | Storage | Scenario | Before (ms) | After (ms) | Change | 95% descriptive bootstrap interval for change | |---|---|---:|---:|---:|---:| | OSS | 100 partitions, SUM | 1,368.7 | 1,374.9 | +0.45% | [-1.15%, +2.91%] | | OSS | 10,000 partitions, SUM with p < 100 | 1,396.1 | 1,399.7 | +0.26% | [-1.54%, +2.23%] | | HDFS | 100 partitions, SUM | 1,023.8 | 1,025.0 | +0.12% | [-6.49%, +5.32%] | | HDFS | 10,000 partitions, SUM with p < 100 | 1,046.7 | 1,029.6 | -1.64% | [-9.18%, +5.31%] | The initial ~4% filtered-SUM slowdown was not consistently reproduced: the sign changed between process orders in the confirmation runs. The intervals include zero. This does not prove zero overhead; the machine is shared, and there are only two FE processes per version in each series. The bootstrap intervals describe the collected query samples and do not capture all process-level or host-level variability. Across the main and confirmation series, **1,120 timed queries and 368 warm-up queries completed successfully**. Pilot queries and the discarded 3 GiB series are outside these totals. A confirmation-run stop-controller failure occurred after that round's queries had succeeded; process exit was checked before resuming the remaining baseline round with the same settings. ### Functional and resource acceptance Acceptance fixture: Hive, local HMS + real OSS, 10,000 partitions x 100 files, common BE 4.1.3. | Acceptance scenario | Result | |---|---| | LIMIT 10 completes normally, 3 repetitions | Background termination after the stop signal: 1 / 113 / 103 ms; maximum **113 ms** | | SUM canceled during active Split enumeration, 3 repetitions | Background termination: 288 / 186 / 118 ms; maximum **288 ms** | | SUM hits a 3-second query timeout, 3 repetitions | Background termination: 313 / 209 / 248 ms; maximum **313 ms** | | First-Split wait configured to 1,000 ms and 2,500 ms | Both waits report the configured deadline and enumeration progress in the error | | Resource release, all 11 acceptance cases | Queue and queued ranges empty; active producers/runs, listeners, and resources zero; termination completed; query callback registry empty | | Requests after stopping | No new observed SDK HTTP dispatch after stop + 10 seconds, with a 12-second observation window | Example first-Split timeout error: ```text Timed out waiting for the first split after 1000 ms: partitions_processed=0/10000, files_enumerated=0, splits_generated=0, active_producers=1 ``` The first-Split timer starts when waiting for the first Split; total SQL wall time also includes earlier planning. The two first-wait cases inject a 5-second SDK delay to exercise cleanup while an SDK call is in progress. Their background termination times after the timeout signal were 4,221 ms and 2,783 ms, both below 10 seconds. An in-flight SDK call can finish after cancellation; this result does not imply instantaneous interruption at the stop signal. The request observer covers the AWS SDK 2.29.52 `UrlConnectionHttpClient.RequestCallable` dispatch to `HttpURLConnection.connect`; internal JDK retries are not separately counted. Hive end-to-end acceptance passed **11/11**. Iceberg was covered by code review and unit tests; equivalent Iceberg cluster acceptance was not performed in this round. ### Other verification - FE unit tests: **476 passed**, zero failures/errors/skips. - FE build through `build.sh --fe`: passed. - Checkstyle and `git diff --check`: passed. Benchmark scripts, data, runtime configuration, and build workarounds are excluded from this PR's source changes. No associated prerequisite PRs were separately backported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
