devinjdangelo commented on code in PR #7141:
URL: https://github.com/apache/arrow-datafusion/pull/7141#discussion_r1283751032
##########
datafusion/core/src/datasource/listing/table.rs:
##########
@@ -804,21 +804,25 @@ impl TableProvider for ListingTable {
.await?;
let file_groups = file_list_stream.try_collect::<Vec<_>>().await?;
-
- if file_groups.len() > 1 {
- return Err(DataFusionError::Plan(
- "Datafusion currently supports tables from single partition
and/or file."
- .to_owned(),
- ));
+ let writer_mode;
+ //if we are writing a single output_partition to a table backed by a
single file
+ //we can append to that file. Otherwise, we can write new files into
the directory
+ //adding new files to the listing table in order to insert to the
table.
+ let input_partitions = input.output_partitioning().partition_count();
+ if file_groups.len() == 1 && input_partitions == 1 {
Review Comment:
> It seems like there should also be a way to write directly to a target
output without having to register a table provider. I am sure there is a way we
just need to come up with a clever API to do so
I definitely agree. I am thinking for that we could use the `COPY TO` SQL
syntax to dump a table to files. We will need to implement a logical plan for
copy to, but it looks like that is already on your radar in #5654. The
execution plan for `CopyTo` is ultimately the same as `InsertExec` which is
currently created by each `FileSink`. Probably want to rename `InsertExec` to
something more generic like `WriteFileSinkExec` at that point.
When that is done we can add options to `DataFrame.write_table` or create
additional methods that allow hooking into a `LogicalPlanBuilder::copy_to`
method to enable writing to files without the need for a registered table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
