devinjdangelo commented on code in PR #7141:
URL: https://github.com/apache/arrow-datafusion/pull/7141#discussion_r1280009542
##########
datafusion/core/src/datasource/physical_plan/mod.rs:
##########
@@ -330,6 +330,8 @@ pub struct FileSinkConfig {
pub object_store_url: ObjectStoreUrl,
/// A vector of [`PartitionedFile`] structs, each representing a file
partition
pub file_groups: Vec<PartitionedFile>,
+ /// Vector of partition paths
+ pub table_paths: Vec<ListingTableUrl>,
Review Comment:
I believe that `file_groups` represents every individual file in every path
contained in the `ListingTable`, whereas `table_paths` is just a list of the
paths themselves. The language in existing comments/variable names is a bit
confusing as it seems we refer to both files and directories of files as
"partitions" sometimes.
I could compute the table_path based on the prefix of individual files in
the table. I'll need to think about this more and do some testing. What if the
listing table does not contain any files yet? Is file_groups empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
