morningman opened a new pull request, #70746:
URL: https://github.com/apache/doris/pull/70746

   ### Background
   
   The frontend (FE) plans external-file byte ranges and assigns each range to 
a backend (BE). Native Parquet and ORC readers can refine a range into physical 
tasks, which remain on that BE. Hive `TABLESAMPLE` selects a subset of the 
ranges planned by FE.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #70664 (introduced the master regressions), #66802 (earlier 
release-branch implementation)
   PR Type: Bugfix
   
   Problem Summary:
   
   Hive sampling can select an entire large file, and coarse synchronous scans 
can leave available BEs idle. Preserve legacy FE sizing for applied samples and 
subdivide eligible ranges before assignment when coarse planning leaves too few 
ranges, within existing count and size budgets. Most of the diff tests 
sampling, actual FE placement and metadata retention.
   
   The FE planning tests use a single 400 MiB file:
   
   | Scenario | Without the fix | With the fix |
   |---|---|---|
   | 4% sample with legacy 16 MiB ranges | 400 MiB selected range | 16 MiB 
selected range |
   | Two eligible BEs with round-robin assignment | 1 BE receives ranges | 2 
BEs receive ranges |
   
   ### Details
   
   - **Change map:** 81 production lines in four files and 273 test lines in 
three files; no generated output. The sampling gate accounts for three 
production lines; bounded placement and metadata-preserving range copies 
account for the other 78.
   - **Frequency:** Reproduced deterministically in FE planning tests with 
coarse splitting enabled, for applied Hive sampling or too few native ranges 
for the available BEs.
   - **Root cause:** The connector receives the coarse hint before FE applies 
sampling. Assignment then treats each remaining coarse range as indivisible, 
while BE refinement only publishes local children.
   - **Why existing tests missed it:** `PluginDrivenScanNodeTableSampleTest` 
supplies prebuilt ranges; sizing and backend-policy tests cover their parts 
separately. The Hive sampling regression allows a full-file sample, and the 
two-level splitting regression checks results without asserting placement.
   - **Fix:** Omit the coarse hint for applied samples. For underfilled 
synchronous plans, split the largest eligible native ranges up to the available 
BE count and `max_file_split_num`, stopping before children fall below the BE 
target. Copies retain connector metadata and proportional scheduling weights; 
planning does not repeat remote file enumeration.
   - **Other paths and branches:** The shared path covers native plugin scans 
and file TVFs, retaining the serial, metadata COUNT, format/capability, 
compression and HTTP gates. Streaming and batch producers retain lazy 
assignment. Branches 4.1/4.2 have analogous placement loss in their legacy 
planners and need adapted backports; their file-level sampling precedes 
splitting. Branches 3.1/4.0 lack coarse FE splitting. No SPI, wire or 
storage-format change is required.
   - **Behavior change:** Underfilled native plans may expose more FE ranges 
and scanning BEs in EXPLAIN/profile output, within the existing budgets.
   
   ### Testing
   
   - Local `run-fe-ut.sh`: 179 tests in 10 related FE/connector classes passed, 
with zero failures, errors or skips. Coverage includes sampling, actual 
assignment/Thrift serialization, range bounds, serial/count exclusions and 
metadata/weight retention.
   - The initial unmodified-master run failed both reproductions. Reverting the 
two fix entry points with the final tests again selected 400 MiB instead of 16 
MiB and assigned the scan to only one BE; restoring them passes.
   - Extended `external_table_p0/hive/test_two_level_file_splits` with 
repeatable PERCENT and ROWS comparisons against its legacy reference. The 
external Hive suite was not run locally.
   
   ### Notes for Reviewers
   
   The fallback reuses already planned ranges and their immutable connector 
descriptors. It adds no remote planning pass or cross-BE child transfer. The BE 
target also bounds the minimum FE child size in this fallback; placement counts 
do not imply a fixed latency improvement.
   
   ### Release note
   
   Preserve Hive sampling granularity and useful multi-BE placement when coarse 
FE splitting is enabled.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Partially verified: the external Hive suite and 
live multi-BE execution were not run.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Code, tests and this description.
   - Human review: not stated
   - Prompts: Analyze 924060929's comments on #70664, then fix both issues from 
the latest master after that PR merged and open a separate PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to