dwsmith1983 commented on issue #6467: URL: https://github.com/apache/datafusion-comet/issues/6467#issuecomment-5922083305
@comphead I looked at q21 in the code and with a small benchmark of its two correlated subquery joins. Spark turns the `EXISTS` and `NOT EXISTS` into LeftSemi and LeftAnti sort-merge joins with the `l_suppkey <>` filter, and Comet runs both natively by default (`spark.comet.exec.sortMergeJoinWithJoinFilter.enabled`). DataFusion evaluates that filter one inner row at a time, so on lineitem-shaped key groups the join takes 8 to 10 times as long as the same join without the filter. Each small key group is also charged a full input batch of memory, so under memory pressure the join can spill once per group. Numbers are in apache/datafusion#25923. My rough estimate puts this at tens of seconds of the gap at SF1000, not all of it, and it doesn't explain the times rising from one iteration to the next. These would help narrow down the rest: - the Comet commit or version the downstream build is based on, since #5731 (Iceberg scans falling back on the `IS NOT NULL` that `explode` pushes down) and #6261 (native plans keeping memory after an early stop) were fixed after 1.0.0 - the q21 physical plan from both runs and the explainFallback output for q21 - spill counts and join time on the two `CometSortMergeJoin` nodes for each iteration - executor memory settings, especially `spark.memory.offHeap.enabled` and its size, and any lost or OOMKilled executors - a run with `spark.comet.exec.sortMergeJoinWithJoinFilter.enabled=false`, which sends only those two joins back to Spark - whether other queries also get slower across their three iterations -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
