devinjdangelo commented on code in PR #7141:
URL: https://github.com/apache/arrow-datafusion/pull/7141#discussion_r1280003732
##########
datafusion/core/src/datasource/listing/table.rs:
##########
@@ -804,21 +804,25 @@ impl TableProvider for ListingTable {
.await?;
let file_groups = file_list_stream.try_collect::<Vec<_>>().await?;
-
- if file_groups.len() > 1 {
- return Err(DataFusionError::Plan(
- "Datafusion currently supports tables from single partition
and/or file."
- .to_owned(),
- ));
+ let writer_mode;
+ //if we are writing a single output_partition to a table backed by a
single file
+ //we can append to that file. Otherwise, we can write new files into
the directory
+ //adding new files to the listing table in order to insert to the
table.
+ let input_partitions = input.output_partitioning().partition_count();
+ if file_groups.len() == 1 && input_partitions == 1 {
Review Comment:
Are you envisioning`ListingTableWriteOptions `as part of the
`ListingOptions` struct (i.e. a property of the registered table itself)? So
the user would do something like:
```rust
ctx.register_csv("csv_table_sink", out_path, CsvReadOptions::default(),
CsvWriteOptions::default().write_mode(WriteMode::Append))
.await?;
```
Subsequent writes to this table would then respect the declared settings on
that table. PartitionBy information is actually already included in
[CsvReadOptions](https://github.com/apache/arrow-datafusion/blob/27a3c81493210435f23b64e8fc65c988bf665579/datafusion/core/src/datasource/file_format/options.rs#L71).
I had been thinking about how to pass this option at write time:
```rust
df.write_table("csv_table_sink", WriteMode::Append)
```
but I was not sure how to propagate that additional argument from the
`LogicalPlan` through to `InsertExec` (or if that even makes sense to do).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
