Peter Toth created SPARK-61534:
----------------------------------

             Summary: Estimate the size of an existence join from its left side
                 Key: SPARK-61534
                 URL: https://issues.apache.org/jira/browse/SPARK-61534
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Peter Toth


The size estimate of a logical join of type {{ExistenceJoin}} is the product of 
the sizes of its two sides. {{SizeInBytesOnlyStatsPlanVisitor.visitJoin}} takes 
the left side's size only for {{LeftSemi}} and {{LeftAnti}}. Every other type 
outside the inner and outer joins falls back to {{default}}, which multiplies 
the sizes of the children. CBO's {{JoinEstimation}} does not support 
{{ExistenceJoin}} either, so it falls back to the same product.

An existence join outputs each row of its left side once, with one boolean 
column added. So its size is at most a little more than the left side's.

{{RewritePredicateSubquery}} plans an existence join for an {{EXISTS}} or 
{{IN}} subquery under an {{OR}}, as in TPC-DS q10, q35 and q45. The joins above 
it see that side as far bigger than it is, and can miss a broadcast.

Taking the left side's size, as for {{LeftSemi}} and {{LeftAnti}}, would fix 
it. It can change the plans of those TPC-DS queries.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to