ngsg commented on PR #5417: URL: https://github.com/apache/hive/pull/5417#issuecomment-2333340798
I have investgated this issue for few days and conclude that the patch is sufficient to solve this issue. This issues happens when a query plan contains a diamond pattern: an operator that have multiple parent operators, and those parent operators have a common ancestor operator. By default, SWO compares and merges the parent operators of a discardable operator. However, it stops gathering and only performs the comparison if it meets an operator with multiple child operators. [1] Therefore, merging 2 identical diamond patterns always leaves an unmerged common ancestor operator. As far as I know, multiple child operators can be produced by only 2 optimizers: DynamicPartitionPruningOptimization and SharedWorkOptimizer. In case of SWO, there cannot exists 2 identical diamond patterns because if they are identical, then they should be merged by SWO previously. So the only possible problem is DPPOpt, which is covered by this patch. The following link contains some notes about this issue. For those who are interested, feel free to check it out. https://gist.github.com/ngsg/ec53cd605ee58d715b17b8449ef5a5e5 [1] https://github.com/apache/hive/blob/e0bd9eac32851b27976a78cd83584cb867b83182/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java#L1695-L1698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
