sunchao commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5399704303
### [P2] Avoid forcing scalar partition subqueries from outputPartitioning The adaptive-broadcast guard fixes the earlier DPP shape, but [this getter](https://github.com/apache/datafusion-comet/blob/6d20ad69fd39f52c35230d2fa5373aa215e7ed5e/contrib/delta-spark/src/main/scala/org/apache/spark/sql/comet/CometDeltaNativeScanExec.scala#L75-L91) still forces `perPartitionData` when a partition filter contains an unresolved `ScalarSubquery` rather than `InSubqueryExec`. With AQE/coalescing enabled, a Delta `fact` table partitioned by `p`, and Parquet `dim`/`other` tables, this shape reaches the failure: ```sql SELECT id, p FROM fact WHERE p = (SELECT max(p) FROM dim WHERE country = 'yes') UNION ALL SELECT cast(count(*) AS int) AS id, p FROM other GROUP BY p ``` A reduced Spark 4.0.2 / Delta 4.0.0 planning harness, using the current production contribution registry and keeping the parent operators on Spark, reproduces: ```text ValidateRequirements -> CometDeltaNativeScanExec.outputPartitioning -> serializedPartitionData / getFilePartitions -> ScalarSubquery.eval -> requirement failed: Subquery ... has not finished ``` The claim has already returned successfully, so the registry's exception fallback cannot catch this. `doPrepare`/execution-time subquery resolution also occurs too late. Stock Spark completes the query; changing only the scratch probe's getter to return non-forcing `UnknownPartitioning(0)` gets past this validation and reaches the custom AQE optimizer. No Comet/JNI reader executed in the probe. Could we keep partitioning inspection from evaluating unresolved scalar partition filters as well, with a regression covering this mixed `UNION ALL`/shuffle shape? This is a remaining case of the [existing AQE P2](https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835076676). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
