dongjoon-hyun commented on code in PR #60079:
URL: https://github.com/apache/spark/pull/60079#discussion_r4469135061
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:
##########
@@ -204,7 +204,16 @@ class DataSourceV2Strategy(session: SparkSession) extends
Strategy with Predicat
// pruning while the FilterExec above the scan re-evaluates them, so the
two evaluations
// may disagree and rows the source pruned away could not be recovered.
This is the
// runtime counterpart of the pushFilters guard in PushDownUtils
(SPARK-58207).
- val scalarSubqueryFilters = if (relation.runtimeFilterAttrs.nonEmpty) {
+ // Nothing is routed for a read of a group-based row-level operation
either. The operation
+ // replaces the groups its scan returns, and its reads share that scan,
so a filter of one
+ // read must not prune it: the negated condition that selects the rows
to copy over would
+ // drop the groups where every row matches. Only the group filter of
+ // RowLevelOperationRuntimeGroupFiltering prunes that scan.
Review Comment:
Thank you for the review and the measured fix, @peter-toth. I confirmed the
DPP case on this PR: with the group filter disabled, `UPDATE ... WHERE dep IN
(SELECT d FROM src)` and the `EXISTS` form write the `software` row twice, also
with the default `InMemoryBatchScan`, and your change fixes them. Since it
seems to date back to group-based UPDATE
([SPARK-43975](https://issues.apache.org/jira/browse/SPARK-43975), Apache Spark
3.5.0) and needs the group filter disabled, I'll file a new JIRA issue for it
and fix it in a separate PR with your change. In this PR, I reworded the
comment and the PR description so that they no longer say only the group filter
prunes the scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
