pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4286938924
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/JoinEstimationSuite.scala:
##########
@@ -466,6 +466,51 @@ class JoinEstimationSuite extends StatsEstimationTestBase {
}
}
+ test("existence join") {
+ // table2 (key-1-2 int, key-2-4 int): (1, 2), (2, 3), (2, 4)
+ // table3 (key-1-2 int, key-2-3 int): (1, 2), (2, 3)
+ // An existence join returns every row of its left child with one extra
Boolean attribute, so
+ // its cardinality is the left child's and its width is the left child's
plus that attribute.
+ // The right child's cardinality must not enter either, as the generic
product estimate would
+ // have it, since that can disqualify a broadcast of an otherwise small
build side.
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val join = Join(table2, table3, ExistenceJoin(exists),
+ Some(EqualTo(nameToAttr("key-2-4"), nameToAttr("key-2-3"))),
JoinHint.NONE)
+
+ val expectedStats = Statistics(
+ // Three rows of the left child, each widened by the Boolean, whose
default size is 1.
+ sizeInBytes = 3 * (8 + 4 * 2 + 1),
+ rowCount = Some(3),
+ // Only the left child's column stats: the right child's describe
attributes that an
+ // existence join does not output.
+ attributeStats = AttributeMap(Seq(nameToColInfo("key-1-2"),
nameToColInfo("key-2-4"))))
+ assert(join.stats == expectedStats)
+ }
+
+ test("existence join with an empty or unestimated right child") {
+ // Only the left child's row count enters the estimate, so a right child
with no row count,
Review Comment:
You are right on both counts, and this was my error. `StatsTestPlan` takes a
required `rowCount: BigInt` and its `computeStats` always returns `rowCount =
Some(rowCount)`, so neither fixture had no row count -- `emptyRight` had 0 and
`noStatsRight` had 2, and only their column statistics were missing. The test
name and comment claimed a case the test did not build.
The test is renamed `existence join with a right child that has no row count
or no statistics` and now builds a `NoRowCountPlan` whose `computeStats` omits
`rowCount` entirely, alongside the zero-row-count, no-column-statistics case.
Checked against the mutation you named: tightening
`rowCountsExist(join.left)` to `rowCountsExist(join.left, join.right)` now
fails this test, and only this test, out of the 44 in `JoinEstimationSuite` and
`BasicStatsEstimationSuite`. Before the change it stayed green, as you said it
would.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
