suxiaogang223 opened a new pull request, #65262:
URL: https://github.com/apache/doris/pull/65262
### What problem does this PR solve?
Iceberg system table scans serialize Iceberg SDK `FileScanTask` objects and
the Java metadata scanner materializes rows from the task schema. Doris did not
pass the actually required system table columns into the Iceberg SDK scan, so
metadata tables such as `$files` and `$data_files` were planned with the full
schema. For files metadata tables, the full schema can include virtual
`readable_metrics` even when the SQL only requests columns such as
`file_size_in_bytes`.
This PR applies SDK top-level column projection only for Iceberg system
table scans. Normal Iceberg data table scans continue to rely on Doris BE scan
range params for column pruning.
### What changed?
- Add system-table-only `TableScan.select(...)` projection in
`IcebergScanNode` before planning the Iceberg SDK scan.
- Skip synthesized/global row id and Iceberg row lineage columns when
building the SDK projection list.
- Add regression coverage for ORC and Parquet Iceberg tables with
`map<boolean, boolean>` columns, verifying projected `$data_files` and `$files`
queries.
### Check List (For Author)
- Test: Build
- `DISABLE_BUILD_UI=ON MAVEN_OPTS='-Xmx8g' FE_MAVEN_THREADS=1
FE_MAVEN_RETRY_THREADS=1 ./build.sh --fe`
- Behavior changed: No
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
