comphead commented on issue #7487:
URL: 
https://github.com/apache/datafusion-comet/issues/7487#issuecomment-6932949529

   I profiled the `bhj` mode of the benchmark above with async-profiler, on 
`main` plus #7655. Same setup: a 1M-row
   broadcast probed by 32 tasks on `local[8]`, so 4 waves, lz4 unless noted. 
The arms are Spark, Comet with #7655 (idle
   timeout 1 s), Comet with `idleTimeout=0` (as `main`), and Comet without 
sharing.
   
   async-profiler 4.6 on macOS, 1 ms interval, 20 runs per arm for CPU and 
another 20 for wall clock. macOS cannot sample
   both in one session. Wall samples cover only the executor task threads, so a 
wall sample is 1 ms of one task thread.
   Each recording is cut into the phases of each run by the stage windows on 
the driver. All numbers are per run.
   
   Phases, in ms on the driver clock (runs with the CPU profiler, the 
wall-profiled runs agree within 5%):
   
   | Arm                              | Planning + dim stage | Driver builds 
the broadcast | Probe stage | Total |
   | -------------------------------- | -------------------: | 
--------------------------: | ----------: | ----: |
   | Spark                            |                 38.0 |                  
     180.5 |        33.6 | 252.1 |
   | Comet, #7655                     |                 26.1 |                  
      72.5 |        44.5 | 143.1 |
   | Comet, `idleTimeout=0` (`main`)  |                 25.0 |                  
      71.2 |        85.5 | 181.7 |
   | Comet, no sharing                |                 23.6 |                  
      69.5 |       101.2 | 194.3 |
   
   Probe stage with #7655, summed over the 32 tasks, in ms:
   
   | Frame                                                                | 
Wall |  CPU |
   | -------------------------------------------------------------------- | 
---: | ---: |
   | `SharedDecode$Entry.await`, the other 7 tasks of the first wave      |  
104 |  0.2 |
   | `SharedDecode$.decode`, the one decode                               |   
15 |   15 |
   | `Native.executePlan`                                                 |  
164 |  155 |
   | ↳ `FilterExecStream`, fact scan and filter on the probe side         |     
 | 51.5 |
   | ↳ `collect_left_input`, the hash table each task builds              |     
 | 46.5 |
   | ↳ `build_batch_from_indices`, join output                            |     
 |   21 |
   | ↳ `ScanExec`, import of the broadcast                                |     
 |   16 |
   | ↳ `ArrayMap` lookups                                                 |     
 |  3.4 |
   | `createPlan`, `releasePlan` and the output import                    |   
12 |   13 |
   
   - Probe-stage CPU on the task threads is about the same in both engines, 212 
ms for Comet and 214 ms for Spark. With
     #7655 the extra wall time is the first wave waiting for the one decode. 
The wait is 104 ms with #7655 and 370 ms
     with `idleTimeout=0`. With zstd level 5 it is 188 ms and 641 ms.
   - The decode and the wait happen while the inputs resolve, before 
`createPlan`. Every wait sample is under
     `CometExecRDD.resolveInputObjects` > 
`CometArrowStream.reconcileStreamSchema`, which reads the first batch of each
     JVM input to build the stream's schema. No metric reports this time. 
"Total time for collecting build-side of join"
     is only the 1.4 ms build per task.
   - The decode is decompression. lz4, 15.1 ms: Java `Unsafe` decompression 
10.5 ms, the `StreamingXXHash32` block
     checksum 3.4 ms, copies 1 ms. lz4-java runs its `LZ4JavaUnsafe*` code on 
this Mac, not the JNI one. zstd level 5,
     26.9 ms: `ZSTD_decompressSequences` alone is 20.1 ms.
   - Each task builds its own hash table: `ArrayMap::try_new` 37.4 ms plus 9 ms 
of min/max over the keys, which
     DataFusion 55.2 collects for the perfect hash path 
(`should_collect_min_max_for_phj`). That is about 1.45 ms per
     task that sharing the table (level 2 of #6013) would save here.
   - Each task validates the UTF-8 of the broadcast's string column again when 
it imports it: `decode_generic_string`
     11.3 ms and `simdutf8` 4.5 ms per run.
   - On the driver, `Utils.coalesceBroadcastBatches` takes 58.6 ms of CPU: 36 
ms compressing the coalesced stream and
     14.8 ms decompressing the 128 batches it reads. The torrent write 
compresses again for 5.9 ms (#2216). Spark's
     driver phase is 103 ms building `LongHashedRelation` and 69 ms in 
`TorrentBroadcast.writeBlocks`, 50 ms of it lz4.
   - With zstd level 5 a run takes 177.6 ms with Comet and 415.8 ms with Spark. 
Comet's driver phase is 99.5 ms there,
     77 ms of it in `coalesceBroadcastBatches`.
   
   Caveats: this is `local[8]` on a shared Mac. In local mode Spark's tasks 
read the driver's own `HashedRelation`, so
   Spark pays no per-executor deserialization here, which a cluster would add. 
The benchmark is still not in the tree.
   Happy to share the flamegraphs per arm and phase.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to