comphead opened a new pull request, #7663:
URL: https://github.com/apache/datafusion-comet/pull/7663

   ## Which issue does this PR close?
   
   Closes #6530.
   
   The skewed-join benchmark from #6528 found it. #6528 stays open.
   
   ## Rationale for this change
   
   With AQE on, Spark's `OptimizeSkewedJoin` splits a skewed partition of a 
sort-merge or shuffled hash join by putting an `AQEShuffleReadExec` between 
each shuffle stage and the join. It runs in AQE's query stage preparation 
rules, before `CometRule`. `CometExecRule` converted a Comet shuffle stage into 
a native input but left the read as it was. So the join, the sort under a 
sort-merge join, and everything above the join in its stage ran in Spark, and 
no fallback reason was recorded. AQE and its skew join handling are on by 
default, so a skewed join that AQE splits lost Comet for its whole stage.
   
   ## What changes are included in this PR?
   
   - `CometExecRule.convertNode` converts an `AQEShuffleReadExec` over a Comet 
shuffle stage into a native shuffle input, the way it converts the stage 
itself. By the time `transformUp` reaches the read, the stage under it is 
already a `CometSinkPlaceHolder`, so the case matches that and converts a read 
over the bare stage. It applies only while AQE prepares a plan, which is the 
only time a read is in place before the operators above it are converted. The 
reads that AQE adds later, for coalescing, local reads and rebalance splits, 
sit under operators that are already native, and that path is unchanged.
   - `CometExchangeSink.shouldUseShuffleScan` accepts that read, so it becomes 
a `ShuffleScan` leaf when shuffle direct read is on, and a `Scan` otherwise.
   
   Execution needs no change. It already treats an `AQEShuffleReadExec` as a 
shuffle input, as it does for coalesced reads, and `CometShuffledRowRDD` reads 
all four partition spec types. The Comet joins take `requiredChildDistribution` 
from the Spark join they replace, which is `UnspecifiedDistribution` for a skew 
join, and compute `outputPartitioning` the way Spark's `ShuffledJoin` does. 
dwsmith1983 described the same conversion in a comment on #6530.
   
   ## How are these changes tested?
   
   - New tests in `CometNativeShuffleSuite`: a join with 90% of the left rows 
on one key and a partial aggregate above it, for the `MERGE` and `SHUFFLE_HASH` 
hints, with partition coalescing on and off, each with shuffle direct read on 
and off. They check that AQE split the partition, that the final plan and every 
query stage plan hold only Comet operators, and that both join inputs read 
their split partitions as native shuffle inputs. Without the change all four 
fail with the plan from the issue, `SortMergeJoin(skew=true)` over 
`ColumnarToRow` over `AQEShuffleRead skewed`.
   - Locally on Spark 4.1: `CometNativeShuffleSuite` (87 tests), 
`CometExecRuleSuite` and `CometJoinSuite` (239 tests) pass, and the Spark 3.5 
strict-warnings build passes. Spark's SQL suites have not run yet. Spark's own 
skew join test in `AdaptiveQueryExecSuite` is ignored for Comet in `dev/diffs`.
   - Benchmark: `CometSkewedJoinBenchmark` from #6528, which is not part of 
this PR, with 90% of the fact rows on one key, Spark 4.1.3, `local[8]`, median 
of 9 rounds, on an Apple M3 Max. Comet's total task time with AQE on, in ms, 
and its speedup over Spark with AQE on in the same run:
   
   | case | main | this PR | speedup over Spark, main → this PR |
   |---|---:|---:|---:|
   | SMJ then aggregate, flat payload | 1047 | 527 | 1.64x → 3.79x |
   | SMJ then aggregate, nested payload | 1981 | 922 | 1.97x → 4.08x |
   | SMJ, skewed side buffered, flat payload | 675 | 582 | 2.65x → 3.20x |
   | SMJ, skewed side buffered, nested payload | 2125 | 1541 | 1.99x → 2.55x |
   | SMJ, skewed side streamed, flat payload | 644 | 531 | 2.70x → 3.57x |
   | SMJ, skewed side streamed, nested payload | 1645 | 1762 | 2.15x → 2.21x |
   | SHJ, skewed side probed, flat payload | 673 | 416 | 2.42x → 3.63x |
   | SHJ, skewed side probed, nested payload | 1634 | 1445 | 2.18x → 2.33x |
   
   On main all eight of these plans fell back under AQE. With this PR none 
does, and turning AQE on no longer adds task time for Comet: its AQE-on time 
divided by its AQE-off time went from 1.06-1.62 to 0.66-1.00. The two builds 
ran one after the other. The broadcast join and window cases, which this PR 
does not change, were up to 10% slower in the run with this PR, so a difference 
of that size between the two columns is noise.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to