xylaaaaa commented on issue #69884: URL: https://github.com/apache/doris/issues/69884#issuecomment-6690705109
## 2026-11-07: Delta-only statement-local scan planning reuse Implemented and locally verified on top of `fb4e695560`. Local commit: `8b5d68c46bc638c0fe35b76d1d312d4def2ed621`, **not pushed yet**. Production changes are confined to four Java files in the Delta plugin; shared FE/SPI/BE, Hive, Iceberg and Paimon are unchanged. The existing `enable_external_scan_task_reuse` flag and planning scope now share completed synchronous ranges and per-node metrics. Keys include catalog/query/user, table coordinates/UUID/version, typed predicate, partition selection, COUNT, split size and time zone. Identical concurrent loads coalesce; waiting callers can cancel independently; failed/cancelled loads are retried. The memo retains at most 128 entries, 16,384 ranges and 16 MiB of estimated range payload. This is a retention budget, not a whole-query/in-flight heap bound. Streaming never materializes a list to enter this memo. ### Validation - Full `build.sh --fe -j4` and Checkstyle passed. BE is unchanged and uses the preceding ASAN build at the same base revision. - **102 distinct executed unit tests passed**, including real provider integration, typed keys, version/scope/query/catalog/user isolation, COUNT/partitions, descriptor ownership, concurrent loading, cancellation/interruption, retry, budgets and million-file streaming. The optional PlanningMetrics diagnostic class was not run. Final test refinements also passed reactor validation/Checkstyle. - Official regression comparison passed: basic 36 blocks, new scan_reuse 27, parquet_deletion_vector 32, mixed HMS/Delta/Iceberg 14: **109 blocks**. The new .out was generated by the runner, independently checked, and rerun without -genOut. Existing .out files are unchanged. - New coverage uses retained data through filesystem, dedicated HMS and HMS gateway, with reuse disabled/enabled and streaming. It checks DV id/payload alignment, metadata/filtered COUNT and column mapping. Each three-branch query records 1 reuse hit / 2 enumerations when enabled and 0 hits / 3 enumerations when disabled or streaming. ### Same-build targeted A/B Retained SF1 data; macOS ARM64 ASAN, four pipeline tasks, 8 GiB query limit, SQL/file cache disabled, UTC. All 24 row counts were ready through the existing default refresh interval before timing. Each setting had three warmups, ten alternating EXPLAIN pairs and seven execution pairs. All 88 Delta executions, including warmups and post-measurement profiles, matched four Hive reference executions. | Query | EXPLAIN median, off → on (ms) | Query median, off → on (ms) | Actual enumerations | Checkpoint range GETs | |---|---:|---:|---:|---:| | Q88 | 323.616 → 152.279 (-52.9%) | 2268.353 → 2050.009 (-9.6%) | 32 → 11 | 160 → 55 | | Q80 | 267.732 → 203.634 (-23.9%) | 3571.101 → 3502.396 (-1.9%) | 18 → 12 | 90 → 60 | | Q4 | 234.263 → 118.486 (-49.4%) | 9228.730 → 9203.800 (essentially flat) | 18 → 6 | 90 → 30 | | Q22 | 56.011 → 50.867 | 9794.371 → 9840.166 (essentially flat, +0.5%) | 3 → 3 | 15 → 15 | Q22 has no reuse hits or IO savings; its approximately 5 ms EXPLAIN difference is not attributed to this optimization. Q88 was faster in all seven execution pairs; Q80 in six, Q4 in five, Q22 in three. Q4's post-measurement profile still spends about 8.8 seconds waiting/fetching execution results, so halving planning time barely changes end-to-end time. Separate MinIO traces, collected **after** timing, give identical counts in all three samples per setting, all HTTP 206. Checkpoint body bytes per EXPLAIN: Q88 3,331,960 → 969,558; Q80 2,329,190 → 1,530,260; Q4 2,360,120 → 793,405; Q22 stays at 338,203. ### Large-table streaming and retention The retained million-file planning fixture was tested in independent 512 MiB JVMs, three complete enumerations each. With FE/BE/Docker stopped on both sides, enumeration median was **1493.940 → 1484.218 ms**; first split **23.286 → 20.804 ms**. Sampled heap peak was **112.921 → 125.744 MB**, so memory is not claimed unchanged. The bounded-heap test passed and no full range list was materialized. An initial after-run with idle services present measured 1552.093 ms; that sample is retained separately rather than discarded. This fixture validates metadata planning, not real million-file data IO. This is a targeted ASAN/SF1 A/B, **not** a full TPC-DS/Hive or RELEASE rerun; the November 6 **+4.69%** remains the latest full-workload result. END-03 and M1 acceptance remain open. README, local design/tasklist and the per-checkpoint self-review are updated. Evidence is retained in `docs/delta-m1-preflight/scan-plan-reuse-20261107/` (report, samples, plans/profiles, sanitized storage traces, JFR, unit-test XML and generated-output audit). All 96 TPC-DS objects match their original path/size/SHA-256. Temporary services are stopped, data/volumes/builds retained, and local configuration changes are not committed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
