andygrove commented on code in PR #6268:
URL: https://github.com/apache/datafusion-comet/pull/6268#discussion_r4185303224
##########
spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:
##########
@@ -1318,6 +1325,9 @@ abstract class CometLeafExec extends CometNativeExec with
LeafExecNode {
* parent's native execution receives an empty input.
(`CometIcebergNativeScanExec` does NOT use
* this trait; it has a dedicated `findAllPlanData` case.)
*
+ * `perPartitionData.length` is the partition count the native block runs
with, and the count in
Review Comment:
Now that `buildNativeContext` sizes every `CometScanWithPlanData` from
`perPartitionData`, could this doc also say that `outputPartitioning` must not
read `perPartitionData`, or anything else that runs the scan's DPP subqueries?
AQE reads it while optimizing a stage, before
`CometPlanAdaptiveDynamicPruningFilters` has converted the placeholders, and
that's the q5 crash. The Delta scan in #5365 copied the old pattern and works
around it with `hasUnevaluableSubqueryFilter`. With the rule written here it
could just report `UnknownPartitioning(0)` like `CometNativeScanExec` does now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
