alamb commented on code in PR #14121:
URL: https://github.com/apache/datafusion/pull/14121#discussion_r1915535246
##########
datafusion/core/src/datasource/physical_plan/mod.rs:
##########
@@ -63,14 +64,53 @@ use crate::{
};
use arrow::datatypes::{DataType, SchemaRef};
+use datafusion_common_runtime::SpawnedTask;
+use datafusion_execution::{SendableRecordBatchStream, TaskContext};
use datafusion_physical_expr::expressions::Column;
use datafusion_physical_expr::PhysicalSortExpr;
use datafusion_physical_expr_common::sort_expr::LexOrdering;
+use datafusion_physical_plan::insert::DataSink;
+use async_trait::async_trait;
use futures::StreamExt;
use log::debug;
use object_store::{path::Path, GetOptions, GetRange, ObjectMeta, ObjectStore};
+/// General behaviors for files that do `DataSink` operations
+#[async_trait]
+pub trait FileSink: DataSink {
+ /// Retrieves the file sink configuration.
+ fn config(&self) -> &FileSinkConfig;
+
+ /// Spawns writer tasks and uses `tokio::join` to collect results.
Review Comment:
It might be helpful to document what a Demux task is here on this trait
##########
datafusion/core/src/datasource/physical_plan/mod.rs:
##########
@@ -90,13 +109,23 @@ pub struct FileSinkConfig {
pub insert_op: InsertOp,
/// Controls whether partition columns are kept for the file
pub keep_partition_by_columns: bool,
+ /// File extension without a dot(.)
+ pub file_extension: String,
Review Comment:
I poked around with this PR and since you need to pass the extension into
the demuxer tasks I didn't see any better way to handle this
So TLDR is I think this is fine and I don't have any better suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
