adriangb commented on code in PR #22026:
URL: https://github.com/apache/datafusion/pull/22026#discussion_r3319225261
##########
datafusion/datasource/src/table_schema.rs:
##########
@@ -140,14 +161,30 @@ impl TableSchema {
}
/// Return a new `TableSchema` with `partition_cols` as its partition
columns,
- /// replacing any existing ones.
+ /// replacing any existing ones. Existing virtual columns are preserved.
#[deprecated(
since = "55.0.0",
note = "use
TableSchema::builder(file_schema).with_table_partition_cols(cols).build()"
)]
pub fn with_table_partition_cols(self, partition_cols: Vec<FieldRef>) ->
Self {
TableSchemaBuilder::new(self.file_schema)
.with_table_partition_cols(partition_cols)
+ .with_virtual_columns(self.virtual_columns)
+ .build()
+ }
+
+ /// Return a new `TableSchema` with `virtual_columns` as its virtual
columns,
+ /// replacing any existing ones. Existing partition columns are preserved.
+ ///
+ /// Virtual columns are produced by the file reader (e.g. a Parquet
+ /// `row_number` column) rather than stored in the files or derived from
+ /// partition paths. Each field must carry an arrow virtual extension type
so
+ /// the reader can recognize it; `ParquetOpener` forwards these fields to
+ ///
`parquet::arrow::arrow_reader::ArrowReaderOptions::with_virtual_columns`.
+ pub fn with_virtual_columns(self, virtual_columns: Vec<FieldRef>) -> Self {
Review Comment:
@mbutrovich is this method necessary? shouldn't callers just use
`TableSchemaBuilder` directly? We deprecated `with_table_partition_cols` so it
seems a bit strange to add a non-deprecated counterpart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
