andygrove commented on PR #5634: URL: https://github.com/apache/datafusion-comet/pull/5634#issuecomment-6015273999
The mixed-stage case of #6454 is fixed in #6459 and merged here. `CometCoalesceShufflePartitions` no longer leaves a stage alone because it holds an `AQEShuffleReadExec`. It hides each read behind a leaf while Spark's `CoalesceShufflePartitions` runs over the whole stage, and puts it back afterwards. Spark's rule then leaves alone the shuffles it would coalesce together with a read one, and coalesces the rest, with the same ancestor sizing and the same `UnknownPartitioning` guard as before. A new `CometExecSuite` test puts a union of a shuffle and a scan below a Cartesian product whose other side is a shuffle. Spark's rule coalesces that side first, and the union's shuffled branch now reads one partition, as with Comet disabled, where the previous revision kept all 10. One difference from Spark remains. A group that already has a read no longer counts when Spark's rule divides the minimum partition count over the stage's groups, so the groups it coalesces can keep more partitions than on Spark, never fewer. The CI run at 14e162b62 failed one test on every Spark version: `filtering ratio policy fallback` in `DynamicPartitionPruning*SuiteAEOn`, in `sql_core-1` and the Hive jobs. Since #6577, Comet's cache scan exposes Spark's own `InMemoryTableScanExec` as its one subquery so that the SQL UI can draw the cached plan. The test caches its dimension table, and the suite's check that every subquery of an adaptive plan contains an `AdaptiveSparkPlanExec` met that scan. The diffs now skip it there. A scratch run of that case on Spark 4.1 planned the pruning filter as an adaptive subquery and returned the right rows, and failed on the check alone. Main is merged in again, including #5859. The cache guide keeps its description of the fused reader and its measurements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
