comphead commented on PR #7655:
URL: 
https://github.com/apache/datafusion-comet/pull/7655#issuecomment-6992919362

   Thanks @andygrove, all three points held up when I checked them against the 
code. Addressed in 51ef4bcf77.
   
   **Memory.** The copies do stack. Each broadcast has its own `SharedDecode`, 
so stage N's idle copy is still alive while stage N+1 decodes and holds its 
own. I measured a query with four broadcast hash joins in consecutive stages: a 
4M-row fact in 32 files, four 1.5M-row dims of 24.4 MB decoded each, 
`local[8]`, AQE off, Spark 4.0. I sampled `CometArrowAllocator` every 
millisecond and took the median of 7 runs with the arms interleaved:
   
   | arm | live shared copies | peak shared bytes | decodes | query time |
   | --- | ---: | ---: | ---: | ---: |
   | `idleTimeout=0` (as `main`) | 1 | 24.4 MB | 13 | 657 ms |
   | `idleTimeout=1s`, no cap | 4 | 97.5 MB | 4 | 556 ms |
   | `idleTimeout=1s`, cap 128 MiB | 4 | 97.5 MB | 4 | 564 ms |
   | `idleTimeout=1s`, cap 50 MiB | 3 | 73.1 MB | 4 | 569 ms |
   
   So without a cap, the timeout adds every broadcast that finished in the last 
second to the executor's peak. The cap was small enough to include here, so the 
PR still closes #7487:
   
   - `spark.comet.exec.broadcastExchange.shareDecoded.maxIdleSize`, 128 MiB by 
default, caps the idle copies of all broadcasts on an executor.
   - A copy larger than the cap is freed as soon as no task holds it, so a 
GB-scale broadcast is never kept idle.
   - Keeping a copy frees the copies that have been idle longest until the rest 
fit. The newest copy is the one the next wave is about to read.
   - The 50 MiB row shows the cap binding: two idle copies plus the held one, 
and each broadcast is still decoded once.
   
   The cap bounds only the idle copies. The copy that running tasks hold is 
unchanged from `main`.
   
   `IdleDecodes` never takes a broadcast's lock. The lock order is always the 
broadcast's lock and then the registry's, and evicted copies are freed outside 
both. While writing the cap I also found that an idle entry freed by another 
path, such as a read with a timeout of 0, could later be closed again by its 
old expiry. That is a double close. An expiry now frees an entry only while it 
is still `current`.
   
   **Tests.** Added "a task that comes after another completes reads its 
decode, which the timer then frees". It runs two tasks one after the other with 
the real expiry thread and a 500 ms timeout, asserts one decode, and waits for 
the thread to free the copy. The cap has its own test with an injected 
`IdleDecodes` and a hand-run scheduler.
   
   **Defaults.** `batches` on both `CometBroadcastValue` and `SharedDecode` now 
takes a `SharedDecode.Retention(idleTimeoutMs, maxIdleBytes)` with no default. 
The tests pass `NoRetention` or an explicit `Retention`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to