kylebrooks-8451 opened a new issue, #362: URL: https://github.com/apache/arrow-datafusion-python/issues/362
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The current `Dataset` `TableProvider` is ugly and converts DataFusion `Expr`s to Pyarrow expressions in large match statements. I propose we move towards using Substrait as the method for translating from DataFusion to / from Pyarrow. This was not available yet when this `TableProvider` was made. **Describe the solution you'd like** Now that both Pyarrow and Datafusion support substrait we could clean up and improve the Pyarrow DataSet `TableProvider` and `ExecutionPlan` by using [pyarrow.substrait](https://arrow.apache.org/docs/dev/python/generated/pyarrow.substrait.run_query.html#pyarrow.substrait.run_query) to execute the scan. **Describe alternatives you've considered** Keep the existing ugly Dataset code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
