morningman opened a new pull request, #70746: URL: https://github.com/apache/doris/pull/70746
### Background The frontend (FE) plans external-file byte ranges and assigns each range to a backend (BE). Native Parquet and ORC readers can refine a range into physical tasks, which remain on that BE. Hive `TABLESAMPLE` selects a subset of the ranges planned by FE. ### What problem does this PR solve? Issue Number: None Related PR: #70664 (introduced the master regressions), #66802 (earlier release-branch implementation) PR Type: Bugfix Problem Summary: Hive sampling can select an entire large file, and coarse synchronous scans can leave available BEs idle. Preserve legacy FE sizing for applied samples and subdivide eligible ranges before assignment when coarse planning leaves too few ranges, within existing count and size budgets. Most of the diff tests sampling, actual FE placement and metadata retention. The FE planning tests use a single 400 MiB file: | Scenario | Without the fix | With the fix | |---|---|---| | 4% sample with legacy 16 MiB ranges | 400 MiB selected range | 16 MiB selected range | | Two eligible BEs with round-robin assignment | 1 BE receives ranges | 2 BEs receive ranges | ### Details - **Change map:** 81 production lines in four files and 273 test lines in three files; no generated output. The sampling gate accounts for three production lines; bounded placement and metadata-preserving range copies account for the other 78. - **Frequency:** Reproduced deterministically in FE planning tests with coarse splitting enabled, for applied Hive sampling or too few native ranges for the available BEs. - **Root cause:** The connector receives the coarse hint before FE applies sampling. Assignment then treats each remaining coarse range as indivisible, while BE refinement only publishes local children. - **Why existing tests missed it:** `PluginDrivenScanNodeTableSampleTest` supplies prebuilt ranges; sizing and backend-policy tests cover their parts separately. The Hive sampling regression allows a full-file sample, and the two-level splitting regression checks results without asserting placement. - **Fix:** Omit the coarse hint for applied samples. For underfilled synchronous plans, split the largest eligible native ranges up to the available BE count and `max_file_split_num`, stopping before children fall below the BE target. Copies retain connector metadata and proportional scheduling weights; planning does not repeat remote file enumeration. - **Other paths and branches:** The shared path covers native plugin scans and file TVFs, retaining the serial, metadata COUNT, format/capability, compression and HTTP gates. Streaming and batch producers retain lazy assignment. Branches 4.1/4.2 have analogous placement loss in their legacy planners and need adapted backports; their file-level sampling precedes splitting. Branches 3.1/4.0 lack coarse FE splitting. No SPI, wire or storage-format change is required. - **Behavior change:** Underfilled native plans may expose more FE ranges and scanning BEs in EXPLAIN/profile output, within the existing budgets. ### Testing - Local `run-fe-ut.sh`: 179 tests in 10 related FE/connector classes passed, with zero failures, errors or skips. Coverage includes sampling, actual assignment/Thrift serialization, range bounds, serial/count exclusions and metadata/weight retention. - The initial unmodified-master run failed both reproductions. Reverting the two fix entry points with the final tests again selected 400 MiB instead of 16 MiB and assigned the scan to only one BE; restoring them passes. - Extended `external_table_p0/hive/test_two_level_file_splits` with repeatable PERCENT and ROWS comparisons against its legacy reference. The external Hive suite was not run locally. ### Notes for Reviewers The fallback reuses already planned ranges and their immutable connector descriptors. It adds no remote planning pass or cross-BE child transfer. The BE target also bounds the minimum FE child size in this fallback; placement counts do not imply a fixed latency improvement. ### Release note Preserve Hive sampling granularity and useful multi-BE placement when coarse FE splitting is enabled. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: the external Hive suite and live multi-BE execution were not run. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests and this description. - Human review: not stated - Prompts: Analyze 924060929's comments on #70664, then fix both issues from the latest master after that PR merged and open a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
