hubgeter opened a new pull request, #70616:
URL: https://github.com/apache/doris/pull/70616

   ### Background
   
   Iceberg `rewrite_data_files` first selects candidate data files, packs them 
into groups by partition, and filters those groups before executing rewrites. 
The maximum accepted file size and the maximum group input size serve different 
purposes.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #56413 (introduced the threshold), #64304 (moved the planner 
into the connector)
   PR Type: Bugfix
   
   Problem Summary:
   
   A single oversized file can be selected as a candidate and then silently 
discarded by group filtering. For example, a 1024-byte file with target size 
512, maximum file size 900 and maximum group size 100000 produces zero groups 
with the default file-count and delete settings. Compare group content against 
the maximum accepted file size so the file reaches the rewrite executor, while 
keeping the group-size setting for bin packing.
   
   ### Details
   
   - **Frequency:** A candidate single-file group exceeds 
`max-file-size-bytes`, stays at or below `max-file-group-size-bytes`, and has 
no delete-related trigger.
   - **Root cause:** `hasTooMuchContent` compares group input against 
`getMaxFileGroupSizeBytes()`. Both count/content alternatives require more than 
one input file, so lowering `min-input-files` to 1 does not repair this case.
   - **Why existing tests missed it:** The existing oversized-file case also 
exceeded the group cap, which made the incorrect comparison succeed.
   - **Fix:** Use `getMaxFileSizeBytes()` for the group-content condition. 
Extend existing planner and procedure tests across both selection layers, using 
validated options, exact input paths, default settings, size boundaries, small 
files and group packing.
   - **Other paths and branches:** Backport to branch-4.2 is recommended. At 
`0a24ed86fb8da943e43facf653b5a3016470177a`, the [legacy planner still uses the 
group-size 
threshold](https://github.com/apache/doris/blob/0a24ed86fb8da943e43facf653b5a3016470177a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlanner.java#L267).
 Its `testFileSizeFiltering` also expects a lone oversized file to be 
discarded; update that expectation and adapt coverage to the legacy constructor 
and action APIs. This PR targets master.
   
   ### Testing
   
   Local validation on 2026-11-12 for head 
`e6393478f39cc09d318de5259f19f58740d3e800`:
   
   - `run-fe-ut.sh --run 
'RewriteDataFilePlannerTest,IcebergProcedureOpsTest,IcebergRewriteDataFilesActionTest'`:
 64 tests passed, zero failures/errors/skips. The same cases fail nine 
assertions before the correction.
   - `build.sh --fe -j4` with UI disabled: passed, including Checkstyle and 
connector architecture gates.
   - Real Iceberg planning over synthetic file metadata exercises both 
selection layers and the procedure boundary. Physical SQL rewriting, backend 
output sizes and performance were not tested.
   
   Publication checks on 2026-11-16: the three affected source/test files are 
unchanged on latest master `f468c7d397ce5c65ad02c62e9f5e0187ba8f950f`, and `git 
merge-tree --write-tree` reports a clean merge. The local commit and its 
validated source are unchanged; tests were not rerun for publication.
   
   ### Release note
   
   Fix Iceberg `rewrite_data_files` skipping a lone file larger than 
`max-file-size-bytes` when it is below the group-size limit. The group-size 
setting continues to control bin packing.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes; oversized singleton groups survive filtering. A 
small singleton is no longer rewritten solely because it exceeds a small 
packing limit.
   - Does this need documentation: Yes; change the group-content threshold in 
the website's rewrite file-selection description to `max-file-size-bytes`.
   - Pre-submission status: Partially verified; local FE build and 64 targeted 
tests passed. Physical SQL/BE rewriting and remote CI remain unverified. 
Branch-4.2 backport tests have not been run.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex.
   - Generated: Code, unit tests and this description.
   - Human review: Pending; the agent performed local self-review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to