gene-bordegaray commented on code in PR #22657:
URL: https://github.com/apache/datafusion/pull/22657#discussion_r3401721653
##########
datafusion/catalog-listing/src/options.rs:
##########
@@ -42,8 +43,10 @@ pub struct ListingOptions {
/// This can add a lot of overhead as it will usually require files
/// to be opened and at least partially parsed.
pub collect_stat: bool,
- /// Group files to avoid that the number of partitions exceeds
- /// this limit
+ /// Group files to avoid that the number of partitions exceeds this limit.
+ ///
+ /// If [`Self::output_partitioning`] is set, its partition count is used
+ /// instead, even when it exceeds this value.
pub target_partitions: usize,
Review Comment:
Addressed in the latest push. Declared output partitioning is now
authoritative: with_output_partitioning aligns target_partitions to the
declared partition count, and scan planning validates the resulting file group
count before advertising the partitioning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
