CuteChuanChuan opened a new pull request, #27133: URL: https://github.com/apache/datafusion/pull/27133
## Which issue does this PR close? - Closes #25808. ## Rationale for this change A `LATERAL` subquery with `SELECT DISTINCT` and a non-equality correlated filter returns duplicate rows. As suggested in the issue discussion, this PR makes the query error instead of returning wrong results (see @alamb 's [comments](https://github.com/apache/datafusion/issues/25808#issuecomment-6479355008)). ## What changes are included in this PR? When the correlated filter is pulled above an `Aggregate`, its column is added to the group keys. For a `DISTINCT`, this only adds duplicate rows. `EXISTS` and `IN` ignore duplicates, so currently `PullUpCorrelatedExpr` skips the `can_pull_over_aggregation` check for every `DISTINCT`. However, `LATERAL` returns those duplicates. This PR skips the check only for a `DISTINCT` under `EXISTS`/`IN`. For `LATERAL`, a `DISTINCT` with a non-equality filter is no longer decorrelated, so the query fails with the same `OuterReferenceColumn` error that the `GROUP BY` form already gives. ## What is the testing strategy for this PR? - `lateral_join.slt`: the issue's query now errors but the equality form still returns correct answer. - `subquery.slt`: `EXISTS` and `IN` with `DISTINCT` and a non-equality filter are still decorrelated and return the right rows. No existing test covered this case. ## Are there any user-facing changes? The `LATERAL` query in the issue now returns a This feature is not implemented error instead of wrong results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
