hu6360567 commented on issue #970: URL: https://github.com/apache/arrow-datafusion/issues/970#issuecomment-943378687
Hi @alamb , The goal of converting LogicPlan back to SQL is to push as much as queries to DBMS, so that the size of result from DBMS can be reduced significantly. Like your proposal, projection/filters/limit can be converted back to SQL to directly execute to DBMS in `scan` implementation. But for more comlex cases, recursive queries on DBMS data source ( query on table of join two tables in same database), `TableProvider` may not be suffcient. My expected workflow: 1. At planner node, Logical Plan is generated by SQL, and is optimized (pushdown, etc). 2. **Split optimized plan into sub plans, which depends on where inputs are located (from datasource or output from other sub plan).** 3. Convert sub plans to SQL, and send sql to each execution node. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
