comphead opened a new pull request, #7655:
URL: https://github.com/apache/datafusion-comet/pull/7655

   ## Which issue does this PR close?
   
   Closes #7487.
   
   Part of #6013.
   
   ## Rationale for this change
   
   #7016 lets the tasks of an executor that read a Comet broadcast at the same 
time share one decoded copy of it, and
   frees the copy when the last of them completes. The tasks of a wave finish 
at about the same time, so the copy is gone
   before the next wave starts, and every wave decodes the broadcast again. 
Every task of a wave also waits for that decode.
   
   Counters added temporarily to `SharedDecode` show this on `main`, with the 
`bhj` mode of a hash-join task-time
   benchmark: a 1M-row broadcast probed by 32 tasks on `local[8]`, so 4 waves, 
lz4, 7 runs. Every run had 32 reads, 4
   decodes and 4 frees. 28 tasks waited about 14 ms each, which is the time of 
the decode they waited for. Each new decode
   started 0.6 to 1.3 ms after the copy before it was freed.
   
   ## What changes are included in this PR?
   
   - A new config, 
`spark.comet.exec.broadcastExchange.shareDecoded.idleTimeout`, 1 second by 
default. When the last task
     holding the decoded copy completes, the executor keeps the copy for that 
long, and a task that starts by then reads
     it instead of decoding the broadcast again. A timeout of 0 frees the copy 
at once, as before.
   - `SharedDecode.release` keeps an entry that no task holds, and schedules 
its expiry on one daemon thread. Each entry
     counts its idle periods, so an expiry scheduled before a task took the 
entry again frees nothing, and an entry is
     never freed while a task holds it. An entry whose decode failed is still 
dropped at once.
   - `CometBroadcastExchangeExec` reads the timeout on the driver, as it does 
`shareDecoded.enabled`, and passes it
     through `CometBatchRDD`.
   - `CometTestBase` sets the timeout to 0. Some suites compare 
`CometArrowAllocator`'s memory before and after a step,
     and the idle copy of an earlier test's broadcast could be freed in between.
   - The config docs, the broadcast join section of the tuning guide, and the 
memory management guide describe the idle
     copy, which is JVM-side Arrow memory that neither Spark nor Comet's pools 
account for.
   
   Mean time per probe task in the same benchmark, median of 7 runs with the 
arms interleaved:
   
   | codec         | Spark  | Comet, this PR | Comet, `idleTimeout=0` (as 
`main`) | decodes per run |
   | ------------- | -----: | -------------: | 
---------------------------------: | --------------: |
   | lz4           | 7.1 ms |         9.0 ms |                            18.8 
ms |          4 to 1 |
   | zstd, level 5 | 7.9 ms |        12.5 ms |                            31.8 
ms |          4 to 1 |
   
   The first wave still waits for its decode. A cheaper decode would shrink 
that wait (#2216, #3762). The idle copy in
   this benchmark is 31.8 MB. The expiry freed it about 1 second after the last 
probe task, and the allocator's memory did
   not grow across runs.
   
   The default of 1 second is a trade-off. The waves were about 1 ms apart 
here, and a cluster adds scheduling delay, but a
   longer timeout keeps memory that no pool accounts for after the stage that 
read it. A cap on the idle bytes per
   executor, as #7487 suggests, is left out to keep the change small.
   
   ## How are these changes tested?
   
   - A new test in `CometBroadcastValueSuite` replaces the scheduler with one 
that the test runs by hand. It checks that a
     task that comes before the timeout reads the copy of earlier tasks without 
decoding it again, that an expiry
     scheduled before that task came frees nothing, while the task runs or 
after it completes, that the expiry of the
     last release frees the copy, and that a timeout of 0 frees it at once.
   - The existing test of broadcast joins over nested columns now runs with a 
100 ms timeout and waits for the expiry
     thread to free the copy, so it covers the real scheduler.
   - The benchmark numbers above. The benchmark is not in the tree yet and will 
come as its own PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to