toutane opened a new issue, #2962: URL: https://github.com/apache/iceberg-rust/issues/2962
### Is your feature request related to a problem or challenge? #2671 introduces eager scan planning for the DataFusion integration. The planned `FileScanTask` are currently assigned to DataFusion output partitions using round-robin. This balances the number of tasks per partition, but not the amount of data to read. When file sizes vary significantly, one partition may receive most of the bytes and become the query bottleneck. This issue concerns distributing already planned tasks between DataFusion partitions. Generic file splitting and scan-task planning remain tracked by #128. ### Describe the solution you'd like Replace the round-robin grouping in `scan_planning.rs` with size-aware grouping. The implementation should: - create at most `target_partitions` non-empty groups; - balance the estimated bytes to read across those groups; - preserve the current behavior for empty task lists and when there are fewer tasks than target partitions; The task weight should represent the bytes scanned by the task, including the case where a task covers only part of a file. Related to #128, but does not replace or close it: #128 covers generic size-based scan-task planning and file splitting in the core Iceberg crate. ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
