cloud-fan commented on PR #58635:
URL: https://github.com/apache/spark/pull/58635#issuecomment-6127914274
Thanks, I confirmed both cases are fixed: an outer generate over an unstable
generator now taints the aggregates above it, and an uncheckpointed LogicalRDD
is rejected. Two gaps of the same kind remain. (1) The Aggregate case clears
multiplicity even when a grouping expression is unstable. With `rnd3` a
nondeterministic UDF returning 0-2, in `SELECT a, count(*) AS c FROM (SELECT a
FROM (SELECT a, rnd3() AS r FROM dim) GROUP BY a, r) GROUP BY a`, `c` is
accepted as a stable key, yet each `a` appears once per distinct `r`, so the
count can differ between the filter's evaluation and the join's. Setting
`unstableMultiplicity = !stableGroups` (or rejecting unstable grouping, as the
class doc says) closes it, with the matching change in the Distinct case. (2)
`DataSourceV2ScanRelation.isOutputRepeatable` is always true, so an unordered
`LIMIT` that a JDBC source fully absorbs passes the leaf check, while the same
LIMIT is rejected when it stays in the plan.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:6058261983","thread_id":"conversation:6058261983","verdict_sha256":"41f2e83c8f4fba6960398fb3dff5399ed0cc1480c5477d4d73b8586c2382b847"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
