comphead commented on issue #4614: URL: https://github.com/apache/datafusion-comet/issues/4614#issuecomment-6755216430
I profiled q72's join in Comet to separate the two candidate causes above. At TPC-DS SF10 locally (Spark 4.1.1 local mode with 8 cores, Comet main on DataFusion 55.2.0), the native sort-merge join itself is most of the cost, rather than the lost fusion: - `CometSortMergeJoin` took 86% of the native join time in its stage, and the eight broadcast joins above it took 14%. - In a CPU profile of the stage, 53% was in DataFusion's `freeze_streamed`, which materializes every column of every candidate pair before it applies the join filter, and 26% was recording pairs. Evaluating `inv_quantity_on_hand < cs_quantity` itself took about 1%, and 95% of the 23.5B candidate pairs fail it. apache/datafusion#27278 filters inner sort-merge join pairs before materializing them. With it backported to 55.2.0, q72 at SF10 took 12.0 to 12.2 s against 43 to 49 s before (Spark: 13.8 s), and the join's own time went from 270 s to 40 s. For context, this has reached users since #4062 enabled `spark.comet.exec.sortMergeJoinWithJoinFilter.enabled` by default. In the data-on-eks TPC-DS 3TB Iceberg results (awslabs/data-on-eks#1182), q72 took 43.1 s with Comet 0.17.0, 134.1 s with 1.0.0 and 119.3 s with 1.2.0, against 55.5 s in Spark 3.5.8. Turning the flag off restores q72, but it also gives up q95's gain, because q95's `<>` self-join keeps 83% of its pairs (0.9 s with the flag on, 2.0 s off and 4.2 s in Spark at SF10). So a blanket revert would be a trade rather than a fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
