alamb commented on code in PR #20188:
URL: https://github.com/apache/datafusion/pull/20188#discussion_r2777550128
##########
datafusion/datasource/src/file_scan_config.rs:
##########
@@ -332,7 +350,12 @@ impl FileScanConfigBuilder {
/// Set the columns on which to project the data using column indices.
///
- /// Indexes that are higher than the number of columns of `file_schema`
refer to `table_partition_cols`.
+ /// This method attempts to push down the projection to the underlying file
+ /// source if supported. If the file source does not support projection
+ /// pushdown, an error is returned.
Review Comment:
Yeah, I agree -- I struggled with fixing everything I found questionable
with just documenting what it was doing
##########
datafusion/datasource/src/file.rs:
##########
@@ -64,24 +70,38 @@ pub trait FileSource: Send + Sync {
) -> Result<Arc<dyn FileOpener>>;
/// Any
fn as_any(&self) -> &dyn Any;
+
/// Returns the table schema for this file source.
///
- /// This always returns the unprojected schema (the full schema of the
data).
+ /// This always returns the unprojected schema (the full schema of the
data)
+ /// without [`Self::projection`] applied.
+ ///
+ /// The output schema of this `FileSource` is this TableSchema
+ /// with [`Self::projection`] applied.
Review Comment:
There is `ProjectionExprs::project_schema` which does so. Maybe I could add
a link to that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
