tustvold opened a new issue, #2291:
URL: https://github.com/apache/arrow-datafusion/issues/2291
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
Part of #2079
Currently the values of Hive partitions are projected within each of the
various file format specific physical plans. As described in #2079 this has a
number of drawbacks
**Describe the solution you'd like**
Rather than handling partition projection within the file scan operator, I
would like to propose modifying `ListingTable` to add a `ProjectionExec` within
`TableProvider::scan` instead of relying on `FileFormat::create_physical_plan`
to do this. This `ProjectionExec` would be created with a set of literal
expressions corresponding to the partition values
Therefore instead of `TableProvider::scan` generating something like
```
AvroExec:
```
It would generate
```
ProjectionExec: ...
AvroExec: ...
```
Note: this will depend on #2289
**Describe alternatives you've considered**
The logic could instead be moved to `FileFormat::create_physical_plan`
implementations, but I think it would be better to keep what is a catalog
detail close to the catalog implementation.
FYI @matthewmturner @yjshen @rdettai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
