hu6360567 commented on issue #970: URL: https://github.com/apache/arrow-datafusion/issues/970#issuecomment-943439585
> but the biggest change would be allowing datasources to declare if they support certain plans. Hi @nevi-me , That's why I convert sub plan back to sql. At planner node, the general planner do not need to know the capability of underlying datasource. At execution node, the executor can construct logical plan from SQL to check if DBMS can execute the query, otherwise, split into more sub plans to directly execute at DBMS, and aggregation operations can be done inside datafusion. In a word, the plan can be optimized as much as possible, if capability is not known; the limit of capability is resolved when it is known. For example, with a recursive architecture, the exeuction node can be a planner of exeuction group recursively. The underlying datasouce capability does not need to exposed to the top planner, bu only the node which is directly connect to it. The phsical plan can be changed at runtime, as long as the logical plan is guaranteed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
