gene-bordegaray commented on code in PR #22607:
URL: https://github.com/apache/datafusion/pull/22607#discussion_r3324271327
##########
datafusion/sqllogictest/src/test_context.rs:
##########
@@ -286,6 +303,212 @@ fn register_strict_schema_provider(ctx: &SessionContext) {
);
}
+//
==============================================================================
+// Range Partitioned Table (sqllogictest-only)
+//
==============================================================================
+
+#[derive(Debug)]
+struct RangePartitionedTable {
+ schema: SchemaRef,
+ partitions: Vec<Vec<RecordBatch>>,
+ range_column_index: usize,
+ split_points: Vec<SplitPoint>,
+}
+
+#[async_trait]
+impl TableProvider for RangePartitionedTable {
+ fn schema(&self) -> SchemaRef {
+ Arc::clone(&self.schema)
+ }
+
+ fn table_type(&self) -> TableType {
+ TableType::Base
+ }
+
+ async fn scan(
+ &self,
+ state: &dyn Session,
+ projection: Option<&Vec<usize>>,
+ _filters: &[Expr],
+ _limit: Option<usize>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ let projected_schema = project_schema(&self.schema, projection)?;
+ let mut source = MemorySourceConfig::try_new(
+ &self.partitions,
+ Arc::clone(&self.schema),
+ projection.cloned(),
+ )?;
+ source =
source.with_show_sizes(state.config_options().explain.show_sizes);
+
+ let output_partitioning =
+ self.output_partitioning(projection, &projected_schema)?;
+ let source = RangePartitionedSource {
+ inner: source,
+ output_partitioning,
+ };
+
+ Ok(DataSourceExec::from_data_source(source))
+ }
+}
+
+impl RangePartitionedTable {
+ fn output_partitioning(
Review Comment:
Ok, I think that the `FileScanConfig::with_output_partitioning(...)` with
the `ListingOptions::with_output_partitioning(...)` is the right move. It will
help us declaring range partitioning on actual file/listing tables so that is
great
I think we can leave the `FileScanConfig` / `ListingTable` work as a
follow-up. There is some things we need to ahndle there I don't think belong in
here.
There is a smaller cleanup where `MemorySourceConfig` could have an
`output_partitioning` builder, which would clean up a lot of the boilerplate,
but after looking at it I don’t think it is good because adding it only to
`MemorySourceConfig` feels like a one-off public API just for this fixture.
I would be ok with moving this to its own module now and clean up with the
right fix after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
