comphead commented on issue #7487:
URL:
https://github.com/apache/datafusion-comet/issues/7487#issuecomment-6932949529
I profiled the `bhj` mode of the benchmark above with async-profiler, on
`main` plus #7655. Same setup: a 1M-row
broadcast probed by 32 tasks on `local[8]`, so 4 waves, lz4 unless noted.
The arms are Spark, Comet with #7655 (idle
timeout 1 s), Comet with `idleTimeout=0` (as `main`), and Comet without
sharing.
async-profiler 4.6 on macOS, 1 ms interval, 20 runs per arm for CPU and
another 20 for wall clock. macOS cannot sample
both in one session. Wall samples cover only the executor task threads, so a
wall sample is 1 ms of one task thread.
Each recording is cut into the phases of each run by the stage windows on
the driver. All numbers are per run.
Phases, in ms on the driver clock (runs with the CPU profiler, the
wall-profiled runs agree within 5%):
| Arm | Planning + dim stage | Driver builds
the broadcast | Probe stage | Total |
| -------------------------------- | -------------------: |
--------------------------: | ----------: | ----: |
| Spark | 38.0 |
180.5 | 33.6 | 252.1 |
| Comet, #7655 | 26.1 |
72.5 | 44.5 | 143.1 |
| Comet, `idleTimeout=0` (`main`) | 25.0 |
71.2 | 85.5 | 181.7 |
| Comet, no sharing | 23.6 |
69.5 | 101.2 | 194.3 |
Probe stage with #7655, summed over the 32 tasks, in ms:
| Frame |
Wall | CPU |
| -------------------------------------------------------------------- |
---: | ---: |
| `SharedDecode$Entry.await`, the other 7 tasks of the first wave |
104 | 0.2 |
| `SharedDecode$.decode`, the one decode |
15 | 15 |
| `Native.executePlan` |
164 | 155 |
| ↳ `FilterExecStream`, fact scan and filter on the probe side |
| 51.5 |
| ↳ `collect_left_input`, the hash table each task builds |
| 46.5 |
| ↳ `build_batch_from_indices`, join output |
| 21 |
| ↳ `ScanExec`, import of the broadcast |
| 16 |
| ↳ `ArrayMap` lookups |
| 3.4 |
| `createPlan`, `releasePlan` and the output import |
12 | 13 |
- Probe-stage CPU on the task threads is about the same in both engines, 212
ms for Comet and 214 ms for Spark. With
#7655 the extra wall time is the first wave waiting for the one decode.
The wait is 104 ms with #7655 and 370 ms
with `idleTimeout=0`. With zstd level 5 it is 188 ms and 641 ms.
- The decode and the wait happen while the inputs resolve, before
`createPlan`. Every wait sample is under
`CometExecRDD.resolveInputObjects` >
`CometArrowStream.reconcileStreamSchema`, which reads the first batch of each
JVM input to build the stream's schema. No metric reports this time.
"Total time for collecting build-side of join"
is only the 1.4 ms build per task.
- The decode is decompression. lz4, 15.1 ms: Java `Unsafe` decompression
10.5 ms, the `StreamingXXHash32` block
checksum 3.4 ms, copies 1 ms. lz4-java runs its `LZ4JavaUnsafe*` code on
this Mac, not the JNI one. zstd level 5,
26.9 ms: `ZSTD_decompressSequences` alone is 20.1 ms.
- Each task builds its own hash table: `ArrayMap::try_new` 37.4 ms plus 9 ms
of min/max over the keys, which
DataFusion 55.2 collects for the perfect hash path
(`should_collect_min_max_for_phj`). That is about 1.45 ms per
task that sharing the table (level 2 of #6013) would save here.
- Each task validates the UTF-8 of the broadcast's string column again when
it imports it: `decode_generic_string`
11.3 ms and `simdutf8` 4.5 ms per run.
- On the driver, `Utils.coalesceBroadcastBatches` takes 58.6 ms of CPU: 36
ms compressing the coalesced stream and
14.8 ms decompressing the 128 batches it reads. The torrent write
compresses again for 5.9 ms (#2216). Spark's
driver phase is 103 ms building `LongHashedRelation` and 69 ms in
`TorrentBroadcast.writeBlocks`, 50 ms of it lz4.
- With zstd level 5 a run takes 177.6 ms with Comet and 415.8 ms with Spark.
Comet's driver phase is 99.5 ms there,
77 ms of it in `coalesceBroadcastBatches`.
Caveats: this is `local[8]` on a shared Mac. In local mode Spark's tasks
read the driver's own `HashedRelation`, so
Spark pays no per-executor deserialization here, which a cluster would add.
The benchmark is still not in the tree.
Happy to share the flamegraphs per arm and phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]