peter-toth commented on issue #23531: [SPARK-24497][SQL] Support recursive SQL query URL: https://github.com/apache/spark/pull/23531#issuecomment-527101943 Reviewers, the current state in this PR focuses on the basic recursive functionality. I've removed some of the features (limit support, multiple anchor/recursive terms, ...) that are not required for a minimal implementation and can these will be addressed in follow-up PRs (cc. @maropu). To avoid the issues came from physical plan reuse for different recursive iterations, each iteration starts off from the logical plan of the recursive part (using the statistics of the previous iteration to generate the best possible plan for the current iteration). This approach resulted that no changes are required in rules like 'ReuseExchange` and in adaptive execution, but comes with a minor change in `QueryExecution`. The reuse of common items between recursive iterations (like non correlated subqueries) are also not handled in this PR. I want to address them in follow-ups, for example subquery reuse between recursive iterations can be easily added based on this other PR: https://github.com/apache/spark/pull/25644 For metrics support I renamed some classes/methods of adaptive query execution (ex. `SparkListenerSQLAdaptiveExecutionUpdate` -> `SparkListenerSQLExecutionUpdate`) and I had to add a bit ugly change (https://github.com/apache/spark/pull/23531/files#diff-abfa38a36bdf86c8947ffa7e23e2a7a7R50) to pass `QueryExection` around. Suggestions and feedback are welcome. cc @gatorsmile @cloud-fan @maryannxue @mgaido91 @viirya @hvanhovell
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
