apilloud commented on issue #24314: URL: https://github.com/apache/beam/issues/24314#issuecomment-1332536076
The first step is to capture the output of the planner. The log line `SQLPlan>` contains the logical plan and `BEAMPlan>` contains the physical plan. If the problem shows up in `SQLPlan>` it is a problem in the Calcite half of planning. We have an explicit list of Calcite planner rules we enable, so it is possible that there are rules that would simplify this graph that we just don't have enabled. There might also need to be a new rule added to Calcite to simplify the plan. If the problem shows up only in `BEAMPlan>` it might be a problem with our conversion from logical to physical plans. Our conversion rules assume that the logical plan is optimal, we do no further optimizations when converting to the physical plan. It may be possible that a single logical node is getting turned into multiple physical nodes that do the same thing. There is some logging configuration you can enable in Calcite to output every planner step, that might be helpful too. Our configuration of calcite planner rules is here: https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
