alamb commented on code in PR #18253:
URL: https://github.com/apache/datafusion/pull/18253#discussion_r2465516484
##########
datafusion/datasource/src/file_scan_config.rs:
##########
@@ -175,9 +177,12 @@ pub struct FileScanConfig {
pub file_groups: Vec<FileGroup>,
/// Table constraints
pub constraints: Constraints,
- /// Columns on which to project the data. Indexes that are higher than the
- /// number of columns of `file_schema` refer to `table_partition_cols`.
- pub projection: Option<Vec<usize>>,
+ /// Physical expressions defining the projection to apply when reading
data.
+ ///
+ /// Each expression in the projection can reference columns from both the
file
+ /// schema and table partition columns. If `None`, all columns from the
table
+ /// schema are projected.
+ pub projection: Option<ProjectionExprs>,
Review Comment:
since this is a pub field, I think it qualifies as an API change so I marked
the PR as such
##########
datafusion/datasource/src/file_scan_config.rs:
##########
@@ -455,6 +464,10 @@ impl FileScanConfigBuilder {
file_compression_type.unwrap_or(FileCompressionType::UNCOMPRESSED);
let new_lines_in_values = new_lines_in_values.unwrap_or(false);
+ let projection = projection_indices.as_ref().map(|indices| {
+ ProjectionExprs::from_indices(indices, table_schema.table_schema())
+ });
Review Comment:
Should we deprecate `with_projection` and add a new function
`with_projection_indices` to help downstream users prepare for the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
