blaginin commented on code in PR #14685:
URL: https://github.com/apache/datafusion/pull/14685#discussion_r1975339518
##########
datafusion/core/src/datasource/physical_plan/file_scan_config.rs:
##########
@@ -345,6 +345,32 @@ impl FileScanConfig {
/// Set the projection of the files
pub fn with_projection(mut self, projection: Option<Vec<usize>>) -> Self {
self.projection = projection;
+ self.with_updated_statistics()
+ }
+
+ // Update source statistics with the current projection data
+ fn with_updated_statistics(mut self) -> Self {
+ let max_projection_column = *self
+ .projection
+ .as_ref()
+ .and_then(|proj| proj.iter().max())
+ .unwrap_or(&0);
+
+ if max_projection_column
+ >= self.file_schema.fields().len() +
self.table_partition_cols.len()
+ {
Review Comment:
removed this now. but answering to the question - this could happen if
`projection` set but `table_partition_cols` isn't yet set (or vice versa).
This whole logic should be much cleaner when we switch to the builder
approach (I want to do a PR on top after this one is merged)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
