alamb commented on code in PR #12593:
URL: https://github.com/apache/datafusion/pull/12593#discussion_r1776029106
##########
datafusion/core/src/datasource/physical_plan/parquet/reader.rs:
##########
@@ -57,9 +58,49 @@ pub trait ParquetFileReaderFactory: Debug + Send + Sync +
'static {
file_meta: FileMeta,
metadata_size_hint: Option<usize>,
metrics: &ExecutionPlanMetricsSet,
- ) -> datafusion_common::Result<Box<dyn AsyncFileReader + Send>>;
+ ) -> datafusion_common::Result<Box<dyn ParquetFileReader>>;
}
+/// [`AsyncFileReader`] augmented with a method to customize how file metadata
is loaded.
Review Comment:
I wonder if we can use one of the (many!) already existing APIs /traits for
loading the metadata. The number of layers of abstraction are already pretty
mind boggling. Adding yet another one seems unweildy. Or maybe we could just
change the code not to call the loader if it has metadata 🤔
##########
datafusion/core/src/datasource/physical_plan/parquet/reader.rs:
##########
@@ -57,9 +58,49 @@ pub trait ParquetFileReaderFactory: Debug + Send + Sync +
'static {
file_meta: FileMeta,
metadata_size_hint: Option<usize>,
metrics: &ExecutionPlanMetricsSet,
- ) -> datafusion_common::Result<Box<dyn AsyncFileReader + Send>>;
+ ) -> datafusion_common::Result<Box<dyn ParquetFileReader>>;
}
+/// [`AsyncFileReader`] augmented with a method to customize how file metadata
is loaded.
+pub trait ParquetFileReader: AsyncFileReader + Send + 'static {
+ /// Returns a [`AsyncFileReader`] trait object
+ ///
+ /// This can usually be implemented as `Box::new(*self)`
+ fn upcast(self: Box<Self>) -> Box<dyn AsyncFileReader + 'static>;
Review Comment:
This method is unfortunate, but I also could not figire out a way without it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
