924060929 opened a new pull request, #69298: URL: https://github.com/apache/doris/pull/69298
### What problem does this PR solve? Issue Number: None Problem Summary: External scans share FIFO workers, can wait for metadata submitted to their own pool, and repeatedly parse file paths and search for more backend candidates than exist. Cold Iceberg scans also load manifests serially. Keep the scheduling and scan hot-path improvements together: query-owned bounded task groups, separate split and metadata workers, cooperative continuations, bounded completion-order manifest prefetch, early cancellation and failure propagation, reusable paths and compact assigned scan envelopes. Preserve per-file format and snapshot-specific partition types without repeated file planning or mutating cached values. On the single-backend four-query warm LIMIT diagnostic, bounding candidate lookup reduced median cumulative backend assignment computation per query from 7.94 ms to 0.88 ms. This is a function-level diagnostic, not an end-to-end SQL speedup claim. Signal each backend consumer on data arrival, completion, cancellation and failure so an empt y terminal request does not wait for the previous 100 ms polling interval. A four-query JFR reproduction identified 78 such delayed EOF requests in 400 SQL executions before the fix. ### Release note External Hive and Iceberg scans can produce splits through bounded fair task groups and stop pending production on query cancellation. Cold Iceberg manifest loads can overlap without waiting for earlier slow loads. Session and FE settings control prefetch and task capacity; split request limits and per-file format semantics are preserved. ### Check List (For Author) - Test: 716 scoped FE/SPI/Hive/Iceberg unit tests and 76 focused terminal-notification tests passed, including 1/2/3/8/32/128 backend candidate limits; clean build.sh --fe build with Checkstyle passed. The final three-PR integration passed 997 scoped tests without skips and all three Hive streaming, Iceberg scheduled-split and table-handle-reuse regression suites. A 400-SQL diagnostic replay found zero delayed EOF requests after the notification fix, compared with 78 before. - Behavior changed: Yes, external scan task scheduling, bounded manifest prefetch and transport compaction. - Does this need documentation: Yes, developer scheduling documentation is included. ### Performance validation Complete SQL medians in milliseconds, matching Linux AArch64 Release BE, single backend, 6 CPUs, warm footer/data cache. Both settings are fixed: `split_batch_size=1`, `split_queue_capacity=4096`. Each single-query result has 40 measured samples in forward/reverse order after identical warmup. Cold denotes catalog/manifest cache invalidation, not a cold JVM or BE data cache. COUNT and SUM results are checked, and post-timing profiles verify footer/page hits. | Scenario | Unoptimized master baseline runs | This PR only | |---|---:|---:| | Hive 1,000 partitions, warm LIMIT | 8.60–9.28 | 9.53 | | Hive 10,000 partitions, warm full scan | 752.84–783.66 | 750.39 | | Iceberg 1,000 manifests, cold LIMIT | 925.44–933.77 | 917.29 | | Iceberg 1,000 manifests, cold full scan | 2109.54–2168.54 | 1423.69 | The independent PR does not remove Iceberg full partition-view construction or BE per-file profile initialization; those benefits belong to separate changes. Warm LIMIT overhead is retained in the results rather than claiming uniform speedups. Backend lookup JMH uses production ConsistentHash and Backend classes, 256 virtual nodes, JDK17/macOS ARM64, two forks and five measured iterations after three warmups. With 2–128 backends the lookup takes 0.21–0.26 microseconds, also with four threads. This is a function benchmark, not multi-backend complete SQL. Candidate order and balancing remain unchanged. Four-query completion wall time (15 groups, each containing four queries): Hive warm LIMIT 14.49 ms versus baseline 12.67–13.03 ms; Hive full scan 62.39 ms versus baseline 69.85–89.55 ms; Iceberg 1,000-manifest full scan 475.07 ms versus baseline 549.00–574.75 ms. The extra warm LIMIT scheduling overhead is a measured tradeoff. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
