comphead opened a new pull request, #7655:
URL: https://github.com/apache/datafusion-comet/pull/7655
## Which issue does this PR close?
Closes #7487.
Part of #6013.
## Rationale for this change
#7016 lets the tasks of an executor that read a Comet broadcast at the same
time share one decoded copy of it, and
frees the copy when the last of them completes. The tasks of a wave finish
at about the same time, so the copy is gone
before the next wave starts, and every wave decodes the broadcast again.
Every task of a wave also waits for that decode.
Counters added temporarily to `SharedDecode` show this on `main`, with the
`bhj` mode of a hash-join task-time
benchmark: a 1M-row broadcast probed by 32 tasks on `local[8]`, so 4 waves,
lz4, 7 runs. Every run had 32 reads, 4
decodes and 4 frees. 28 tasks waited about 14 ms each, which is the time of
the decode they waited for. Each new decode
started 0.6 to 1.3 ms after the copy before it was freed.
## What changes are included in this PR?
- A new config,
`spark.comet.exec.broadcastExchange.shareDecoded.idleTimeout`, 1 second by
default. When the last task
holding the decoded copy completes, the executor keeps the copy for that
long, and a task that starts by then reads
it instead of decoding the broadcast again. A timeout of 0 frees the copy
at once, as before.
- `SharedDecode.release` keeps an entry that no task holds, and schedules
its expiry on one daemon thread. Each entry
counts its idle periods, so an expiry scheduled before a task took the
entry again frees nothing, and an entry is
never freed while a task holds it. An entry whose decode failed is still
dropped at once.
- `CometBroadcastExchangeExec` reads the timeout on the driver, as it does
`shareDecoded.enabled`, and passes it
through `CometBatchRDD`.
- `CometTestBase` sets the timeout to 0. Some suites compare
`CometArrowAllocator`'s memory before and after a step,
and the idle copy of an earlier test's broadcast could be freed in between.
- The config docs, the broadcast join section of the tuning guide, and the
memory management guide describe the idle
copy, which is JVM-side Arrow memory that neither Spark nor Comet's pools
account for.
Mean time per probe task in the same benchmark, median of 7 runs with the
arms interleaved:
| codec | Spark | Comet, this PR | Comet, `idleTimeout=0` (as
`main`) | decodes per run |
| ------------- | -----: | -------------: |
---------------------------------: | --------------: |
| lz4 | 7.1 ms | 9.0 ms | 18.8
ms | 4 to 1 |
| zstd, level 5 | 7.9 ms | 12.5 ms | 31.8
ms | 4 to 1 |
The first wave still waits for its decode. A cheaper decode would shrink
that wait (#2216, #3762). The idle copy in
this benchmark is 31.8 MB. The expiry freed it about 1 second after the last
probe task, and the allocator's memory did
not grow across runs.
The default of 1 second is a trade-off. The waves were about 1 ms apart
here, and a cluster adds scheduling delay, but a
longer timeout keeps memory that no pool accounts for after the stage that
read it. A cap on the idle bytes per
executor, as #7487 suggests, is left out to keep the change small.
## How are these changes tested?
- A new test in `CometBroadcastValueSuite` replaces the scheduler with one
that the test runs by hand. It checks that a
task that comes before the timeout reads the copy of earlier tasks without
decoding it again, that an expiry
scheduled before that task came frees nothing, while the task runs or
after it completes, that the expiry of the
last release frees the copy, and that a timeout of 0 frees it at once.
- The existing test of broadcast joins over nested columns now runs with a
100 ms timeout and waits for the expiry
thread to free the copy, so it covers the real scheduler.
- The benchmark numbers above. The benchmark is not in the tree yet and will
come as its own PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]