asolimando commented on PR #24456: URL: https://github.com/apache/datafusion/pull/24456#issuecomment-5373110268
The benchmark results are very promising and I am very excited to see work in this area! I haven't had time to look into the details of the code, but since this is still a draft I'd like to mention [DpHyp](https://www.researchgate.net/publication/47862092_Dynamic_Programming_Strikes_Back) for the join ordering problem. It has been integrated in [CALCITE-6846](https://issues.apache.org/jira/browse/CALCITE-6846), an implementation ported from [Apache Doris](https://doris.apache.org/) with a few additional improvements. According to https://www.alibabacloud.com/blog/602899 it's also the join ordering implementation of both DuckDB and MySQL. DuckDB is especially relevant as it's often cited here in the community as a source of inspiration. DPHyp might be a good alternative over the in-house implementation, especially for dealing with many non inner-join cases, which are always tricky. Alternatively, it might at least provide ideas for some of the open questions we have. I feel that the algorithm itself wouldn't need customization points, as it's mostly cost-agnostic, what we would like to provide is a customization around statistics/cost-model. For the former https://github.com/apache/datafusion/pull/23651 will probably suffice (paired with the improved version of https://github.com/apache/datafusion/pull/21122 which is next on my list). For the latter, I think it can be done via a physical planning rule "costing" alternatives coming from the hypergraph, to pick the cheapest, as physical rules are already a known customization point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
