hubgeter opened a new pull request, #70616: URL: https://github.com/apache/doris/pull/70616
### Background Iceberg `rewrite_data_files` first selects candidate data files, packs them into groups by partition, and filters those groups before executing rewrites. The maximum accepted file size and the maximum group input size serve different purposes. ### What problem does this PR solve? Issue Number: None Related PR: #56413 (introduced the threshold), #64304 (moved the planner into the connector) PR Type: Bugfix Problem Summary: A single oversized file can be selected as a candidate and then silently discarded by group filtering. For example, a 1024-byte file with target size 512, maximum file size 900 and maximum group size 100000 produces zero groups with the default file-count and delete settings. Compare group content against the maximum accepted file size so the file reaches the rewrite executor, while keeping the group-size setting for bin packing. ### Details - **Frequency:** A candidate single-file group exceeds `max-file-size-bytes`, stays at or below `max-file-group-size-bytes`, and has no delete-related trigger. - **Root cause:** `hasTooMuchContent` compares group input against `getMaxFileGroupSizeBytes()`. Both count/content alternatives require more than one input file, so lowering `min-input-files` to 1 does not repair this case. - **Why existing tests missed it:** The existing oversized-file case also exceeded the group cap, which made the incorrect comparison succeed. - **Fix:** Use `getMaxFileSizeBytes()` for the group-content condition. Extend existing planner and procedure tests across both selection layers, using validated options, exact input paths, default settings, size boundaries, small files and group packing. - **Other paths and branches:** Backport to branch-4.2 is recommended. At `0a24ed86fb8da943e43facf653b5a3016470177a`, the [legacy planner still uses the group-size threshold](https://github.com/apache/doris/blob/0a24ed86fb8da943e43facf653b5a3016470177a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/rewrite/RewriteDataFilePlanner.java#L267). Its `testFileSizeFiltering` also expects a lone oversized file to be discarded; update that expectation and adapt coverage to the legacy constructor and action APIs. This PR targets master. ### Testing Local validation on 2026-11-12 for head `e6393478f39cc09d318de5259f19f58740d3e800`: - `run-fe-ut.sh --run 'RewriteDataFilePlannerTest,IcebergProcedureOpsTest,IcebergRewriteDataFilesActionTest'`: 64 tests passed, zero failures/errors/skips. The same cases fail nine assertions before the correction. - `build.sh --fe -j4` with UI disabled: passed, including Checkstyle and connector architecture gates. - Real Iceberg planning over synthetic file metadata exercises both selection layers and the procedure boundary. Physical SQL rewriting, backend output sizes and performance were not tested. Publication checks on 2026-11-16: the three affected source/test files are unchanged on latest master `f468c7d397ce5c65ad02c62e9f5e0187ba8f950f`, and `git merge-tree --write-tree` reports a clean merge. The local commit and its validated source are unchanged; tests were not rerun for publication. ### Release note Fix Iceberg `rewrite_data_files` skipping a lone file larger than `max-file-size-bytes` when it is below the group-size limit. The group-size setting continues to control bin packing. ### Check List (For Author) - Behavior changed: Yes; oversized singleton groups survive filtering. A small singleton is no longer rewritten solely because it exceeds a small packing limit. - Does this need documentation: Yes; change the group-content threshold in the website's rewrite file-selection description to `max-file-size-bytes`. - Pre-submission status: Partially verified; local FE build and 64 targeted tests passed. Physical SQL/BE rewriting and remote CI remain unverified. Branch-4.2 backport tests have not been run. ### AI Disclosure - Tool: OpenAI Codex. - Generated: Code, unit tests and this description. - Human review: Pending; the agent performed local self-review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
